Unlock Secret GovCloud Power: Databricks Security Revealed!

Good afternoon everyone. Appreciate there are a lot of sessions. Appreciate for joining today. Today we are going to talk about how to securely run databricks on AWS for public sector and highly regulated industries. Right. Public sector and highly regulated industries require stringent regulatory requirements and security. So we will be covering about our AWS Go cloud and how databricks fits in within that and some of the compliances you need, you know you can set up and how can you onboard databricks on AWS with that? My name is Venkat Visanathian. I'm a global partner solutions architect managing databricks. Along with me pal Patel, she's a public sector partner solutions architect. She's going to deep dive on the public sector regulatory requirements and Fedramp High and some of those things. So as I mentioned the stringent data regulations and industry standard is is important for the public sector. In order to qualify or run some of the operations, business operations using the cloud databricks and AWS services you need like Fedramp certification or NISD certifications and PCI DSS HIPAA regulatory compliance and so on. And mainly for public sector looking at, you know if you want to qualify for Fedramp High you need to meet 421 security controls and AWS is Fedramp High certified and main security we meet all the 421 security controls. Furthermore like if you do the reason why they put this control is if you do not have this control and like strong encryption or song access controls, even the perimeter protection like auditing capability, if all those things are not available there is the possibility of, you know, today's world there is like a ransomware attack and malware attack or data leakage and so on happens which can damage the reputation of your organization and of the people. Right? So that's the reason why we have this Fed up High, you know requires 421 security controls. With that we can confidently say that AWS is one of the most secured cloud provider. Now I'd like to like Paul Patel. You know she's a public sector specialist so she's going to Talk about how GovCloud and AWS Gow Cloud, how it meets and what are the things that it covers.

Thank you. Thanks Venkat. I hope everyone can hear me as well. Okay, thank you. So yeah, as Venkat said that we do cover a lot of authorizations and compliance programs with our AWS GOW Cloud region. So what is GOW Cloud? Let me summarize in single line. If I have to Say it's an isolated region, isolated infrastructure for infrastructure as a service and we provide all the AWS services with it and it is for the customers with strict compliance and regulation requirements. And you know it is built with the same high security standards that our other AWS regions are built with. But it also specifically addresses US Federal government requirements, the stringent data requirements and controls for all the authorizations that we support. And this is to protect the controlled unclass information, CUI data, the personally identifiable data, export control data and much more such sensitive information. And just to speak about or speak about the Some of the notable authorizations that we support are FedRamp High or DoDCC SRG IL4 and IL5, the NIST 853 as well as 171 the DFARS, HIPAA CMMC level 1 and 2 with level 3 in session currently in progress. So as you see we have like comprehensive list of compliances that caters to a lot of public sector and highly regulated customers. And the origin of our Go Cloud region, it dates back almost a decade ago in 2011 when we had our first region opened up and it was specifically conceived because the US Public customers, specifically the US public sector customers became interested in how they can have their workloads migrated to cloud and how they can architect their cloud compliance at scale and innovate at scale. So just to give you a brief introduction about the two regions, there are two Go cloud regions, US west one and US east one and these regions are there for meeting your high resiliency demands, any of the latency, sensitivity or disaster recovery needs. Also the regions if you notice they are bicoastal so they are separated by the Rocky Mountains and the Greater Plains. So we also thought about the impact of the terrestrial events which is just kind of pretty huge point. Talking about some of the distinguishing features that we have for AWS PowCloud regions. It's a community cloud and you have vetted access to these accounts and it's managed by US citizens on US soil and we support FIPS 142 cryptographic modules or the endpoints for the services that we support on these regions. And we give you the DoD Cloud Access Point connectivity. The data network and machine is isolated from other regions as well as for the two regions. So you have entire separate stack for identity and access management that we provide for our commercial regions. So you have also dedicated AWS management console with a separate and dedicated password credentials that you set up to log into these accounts. Just to give you a brief of how many services we have available. Sometimes we may lag behind in some of the services that are available on commercial cloud. We are pretty consistent in getting there. Beginning with first two columns showing you the services that we have on our two different regions and you can see we have 100/fedramp high authorizations as well as 100 DOD SRG IL5 authorizations with just under 100 for IL4 and IL5 both and we are one of the clouds with most amount of FedRamp High and DOD SRG certifications on among other CSPs. To give you the brief on we do cater to it's not just public sector when we talk about GOW Cloud it caters to other ranges of customers as well. So you can see government agencies at federal, state and local level as well as profit nonprofit orgs, research organizations, educational SLG, telecom system integrators, etc. So it's a wide range that it caters to. With those foundations in place, I wanted to say that databricks can now be deployed on AWS cloud and it is currently in progress to achieve its agency authority to operate for FedRamp High and concurrently pursuing for DoD Impact Level 5 as well to and the main point is both of these authorizations or certifications they are inheriting the AWS's base compliance, the Comprehensive Security and Compliance control. So it's Venkat mentioned for any such authorizations you have to meet like huge number of controls, they are pretty strict, we have to follow it to the word. So AWS does helps with that initial base foundation on which the partners can build this solution. And it's not just those controls that you inherit. We also give you programs and we give you tools and resources to help you accelerate your journey to get the compliances for yourself as well once you start using the regions of or deploying the region. To give you an idea on some of the features that are available for databricks on aws. Kowcloud the overall platform is in currently in public preview so it is available for everyone to use but not ga yet and notable features as you can see are unity catalog Databricks, SQL dashboards, MLflow experiments with all the latest runtimes that you'll have available on commercial also have available here. These are some of the features that are not going to be available in first iteration of authorization, but please note that we are making continuous improvements and so it is work in progress so they will be there soon. With that I wanted to talk about some of the security features that are necessary and that kind of helps you with the Google cloud deployment of databricks. So the first one I wanted to start with is customer managed vpc. So you would have seen in commercial if you deploy the databricks it can be deployed in databricks created and configured AWS VPC the cluster. But on gouache cloud you would deploy it in your managed VPC on your AWS account. And it gives you several benefits to that. And for instance it gives you increased control so you have more control over the network configuration of this vpc allowing you to comply with specific, you know, requirements that your compliance or security team or CISO team is expecting. It gives you lower privilege level. So DataBricks across account IAM role won't need to have that much access to your AWS account or your resources. Simplified network connectivity since it is your VPC you can plug it to your centralized networking. However you have set it up for your AWS centralized base networking within your control tower setup. You limit the outgoing connection to only the sources or data sources that you know of. You put in firewalls proxy in front of that. So you're also limiting the databricks workload running on your account to just have and communicate with the known sources. Next one we have is private private link. Customer managed VPC is kind of a requirement to set up the private link. It is powered by AWS privatelink features. And if you don't know about AWS private link it gives you the private connectivity between your VPC and AWS resources or your on prem and AWS services so you don't have to go over the Internet and the data stays within the AWS network. So with that databricks built upon that to give you the private link it's two type of connections front end through which you can have secure access to databricks front end the interface that you interact with the rest APIs that you may use with the databricks or even the users on your on prem network can securely connect to your databricks front end. Through this you just need to hook into a transit VPC that you might have in your already established AWS centralized networking model. The second is secure connectivity to web application backend. So the actual compute resources of databricks that speaks to the control plane on sorry compute resources on your account for databricks running your databricks runtime or workload or cluster those need to talk to databricks control services as well. And we provide a secure connectivity for that connection. So even that Data is not leaving your AWS network stays within it and you are secured from all fronts. Next is data encryption. This is a very basic requirement and one of the main AWS services that powers this is AWS key management service kms. You might have already heard of it. So databricks of course leverages that to provide you data at REST and in transit encryption and you can actually create a customer managed key so you have control over the lifecycle of the key that you are using to encrypt your data. You can create two type of keys. One keys to secure your all the management related things. For instance your notebooks or databricks, SQL queries, outputs the git integration or personal access token and such. The other you can use to actually secure your data or encrypt your data address which is stored in your S3 bucket or EBS volume, whichever storage you're using. The second portion of it is an in transit data encryption. So the cluster nodes within the same cluster, the communication between them is also encrypted. It is encrypted using an INIT script that you just plug in during when you are first standing up your cluster and that gives you the SSL or TLS connection. Next is enhanced security and compliance module. This is like an add on that databricks has and it is mandatory for your go cloud deployment. It comprises of two portions. First is enhanced security monitoring and that gives you these added features of hardened image. So you have CIS level benchmarked image. It gives you FIPS142 level one validated encryption, the antivirus monitoring agent or file integrity to check the none of your files are changed or if they are changed when and by whom. The regular vulnerability scanning. The second portion of it is compliance security profile. ESM is actually subset of csp. So this is part of just one module that you enable for your go cloud deployment. It is required to give you the regulated work running the regulated workloads like FedRamp I or DoD IL5. It enforces ESM. It also enforces the use of AWS Nitro instances. It's an instance type that uses that extra nitro chip and offloads the compute and the storage from your actual hardware to give you added security. The last part is authentication and access controls. Single sign on. Again, it's a mandatory that you have to set up single sign on. It can be through any of your identity provider. If you're on aws, you can use our identity service. You can provision or users and groups. You can also sync it with your current SCIM we give you secure API authentication so you can have personal access token and you get granular access control systems as well for both workspace level as well as the account level. So just to give you the overview of the features or security features that we saw next up it's data governance features that are sort of differentiated features available for databricks on go cloud and Venkat is gonna speak about that.

Thank you so much. So data governance databricks on data governance mainly on AWS we have something called Unity Catalog already Paul did mention like it is in public preview the Unity catalog is only available on databricks on AWS platform. It is not available on other cloud, Azure, Public Cloud or gcp. So what it means is now using databricks on AWS you get to use the completely lineage access control. And Unity catalog also comes with the data lineage data sharing which is you can share data between two different like multiple workspaces or multi between data or third party. Also you can do it and also enables you to do search and discovery on the reserve whatever Unity catalog features that you see, the complete feature is available on the cloud also on the public cloud. And Delta Live Table is a declarative way of doing things like instead of focusing on how to orchestrate your workflow, what you're focusing is on how do I do transformation. And in a single notebook you can do all the transformation and the Delta Live table is going to find out what is the dependency and it's going to orchestrate your workflow and you can also schedule it. It's going to manage your cluster automatic cluster management and you can put data quality enforcement, you can see see that in Delta Table from the ingestion for every flow you can see how the data quality has taken place and all the metrics is available. So it is basically it is freeing you from thinking about okay, like now I'm adding this table. Whereas like how do I get I need to change the workflow. Now those things you don't need to do that. It's going to take care of itself. And with Unity Catalog you can do secured data sharing. And when you say data sharing it is not just the data sharing. You can share the data associated notebook models and any assets that is managed within Unity Catalog you can share with other agencies within PUB sec. So these features are not available in or it is available only on aws. Now if you want to open an account in AWS KAO cloud you just sign into standard management console and you sign up. But before you do that, you also need to have a standard account. So the standard account is where all the billing information will be available. So you need to have two accounts. Once you have this two account setup, then to set up databricks, you will reach databricks directly. There is no marketplace offering it for for go cloud. Thank you so much. Thank you PA Thank.