{"id":39171,"date":"2020-08-20T15:00:21","date_gmt":"2020-08-20T14:00:21","guid":{"rendered":"https:\/\/cm-edgetun.pages.dev\/en-gb\/industry\/blog\/?p=39171"},"modified":"2020-08-27T13:31:38","modified_gmt":"2020-08-27T12:31:38","slug":"secure-access-to-storage-azure-databricks-and-azure-data-lake-storage-gen2-patterns","status":"publish","type":"post","link":"https:\/\/cm-edgetun.pages.dev\/en-gb\/industry\/blog\/technetuk\/2020\/08\/20\/secure-access-to-storage-azure-databricks-and-azure-data-lake-storage-gen2-patterns\/","title":{"rendered":"Secure Access to Storage: Azure Databricks and Azure Data Lake Storage Gen2 Patterns"},"content":{"rendered":"

\"The<\/p>\n

There are a number of considerations when configuring access to Azure Data Lake Storage gen2 (ADLS) from Azure Databricks (ADB). How will Databricks users connect to the lake securely, and how does one configure access control based on identity? In a previous article<\/a> we covered six access control<\/a> patterns, the advantages and disadvantages of each, and the scenarios in which they would be most appropriate. This article aims to complete the security discussion by providing an overview of network security<\/a> between these two services, and how to connect securely to ADLS from ADB using Azure Private Link.<\/p>\n

 <\/p>\n

Secure access to Storage\/ADLS Gen2<\/h2>\n

In Azure there are two types of PaaS service \u2013 those which are built using dedicated architecture, known as dedicated services, and those which are built using a shared architecture, known as shared services. Dedicated services use a mix of cloud resources (compute, storage, network) allocated from a pool, and are assigned to a dedicated instance of that service for a particular customer. These can be deployed within a customer virtual network, for example, a virtual machine. Shared services use a set of cloud resources which are assigned to more than one instance of a service, utilised by more than one customer, and therefore cannot be deployed within a single customer network e.g. storage. Depending on the type of service, a different VNet integration pattern<\/a> is applied to make it accessible only from clients deployed within Azure VNets and not accessible from the internet.<\/p>\n

Azure Storage \/ ADLS gen2 is a shared service built using a shared architecture, and so to access it securely from Azure Databricks there are two options available. This Databricks blog<\/a> summarises the following approaches:<\/p>\n

    \n
  1. Service Endpoints<\/a><\/li>\n
  2. Azure Private Link<\/a><\/li>\n<\/ol>\n

    Customers may use either approaches for securing access between ADB and ADLS Gen2, but both require the ADB workspace to be VNet injected<\/a>.<\/p>\n

    Service Endpoints<\/h3>\n

    The documentation<\/a> explains how to configure service endpoints, and how to limit access to the storage account by configuring the storage firewall. Further secure the storage account from data exfiltration using a service endpoint policy<\/a>.<\/p>\n

    Private Link<\/h3>\n

    The setup for storage service endpoints are less complicated than Private Link, however Private Link is widely regarded as the most secure approach and indeed the recommended mechanism for securely connecting to ADLS G2 from Azure Databricks. It exposes the PaaS shared services (storage) via a private IP and thus overcomes the limitations of service endpoints and protects against data exfiltration by default. The setup of Private Link requires a number of configurations at the network and DNS level and the complexity encountered is around the DNS resolution to the service. The following article<\/a> goes into greater detail on DNS considerations and integration scenarios. The approach discussed below is to use Azure Private DNS Zones to host the \u201cprivatelink\u201d zone.<\/p>\n

    Connecting securely to ALDS from ADB<\/h3>\n

    The following steps will enable Azure Databricks to connect privately and securely with Azure Storage via private endpoint using a hub and spoke<\/a> configuration i.e. ADB and private endpoints are in their respective spoke VNETs:<\/p>\n

      \n
    1. Deploy Azure Databricks into a VNet using the Portal<\/a> or ARM template<\/a>.<\/li>\n
    2. Create a private storage account<\/a> with a private endpoint and deploy it into the different VNet (i.e. create a new VNet named spokevnet-storage-pl beforehand).<\/li>\n
    3. Ensure the private endpoint is integrated with a private DNS zone<\/a> to host the privatelink DNS zone of the respective service, in this case dfs.core.windows.net. When creating the Private Endpoint, there is an option to integrate it with Private DNS as shown below:\"a<\/li>\n
    4. When ADB and Storage private endpoints are deployed in their respective VNets, there are some additional steps that need to be performed:<\/span>\n