![\"An](\"https:\/\/cm-edgetun.pages.dev\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/02\/databricksheader.jpg\")
<\/p>\n<\/p>\n
![\"An](\"https:\/\/cm-edgetun.pages.dev\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/08\/JiTAD1.png\")
<\/p>\n
Using AAD tokens it is now possible to generate an Azure Databricks personal access token programmatically, and provision an instance pool using the Instance Pools API. The token can be generated and utilised at run-time to provide \u201cjust-in-time\u201d access to the Databricks workspace. Using the same AAD token, an instance pool can also be provisioned and used to run a series of Databricks activities in the same ADF pipeline.<\/p>\n
For those orchestrating Databricks activities via Azure Data Factory, this can offer a number of potential advantages:<\/p>\n
<\/p>\n
The following diagram depicts the architecture and flow of events:<\/p>\n
![\"An](\"https:\/\/cm-edgetun.pages.dev\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/08\/JiTAD2.png\")
<\/p>\n
Extending this approach a little further can provide excellent separation of concerns<\/a> between the platform team responsible for provisioning the infrastructure i.e. the Databricks runtime environment, and the data team depending on this environment to run their data pipelines. Using the technique described in this blog<\/a>, it would be possible for the platform team to manage an \u201cinitialisation\u201d pipeline which takes care of provisioning the environment as well as any validation and repeatable business logic. This \u201cenvironment initialisation\u201d pipeline may run in the same or another Data Factory, which then invokes or is invoked by the engineering pipeline managed by the data team running the Databricks workloads.<\/p>\n <\/p>\n The following demo will provide a step-by-step tutorial to setup a Function app to generate the token and provision the pool, and an ADF pipeline which is provided just-in-time access to the workspace at run-time, leveraging cluster pools to run a series of Databricks activities.<\/p>\n Note: Any code provided should not be regarded as production ready but is simply functional for demonstration purposes.<\/strong><\/p>\n <\/p>\n If you wish to complete this demonstration you will need to provision the following services:<\/p>\n Additional steps:<\/p>\n <\/p>\n As per the steps in the repo ensure that the service principal has been granted access to the secrets in Key Vault via an access policy.<\/p>\n In a production scenario one would need at least two Key Vaults, one for the Platform team to store their secrets that will be used by the Function App, and another for the Data team to store Databricks tokens and cluster pool IDs. For the purposes of this demo only one Key Vault is required.<\/p>\n <\/p>\n After running the setup scripts and publishing the code to the function app, there is no further configuration required. The setup script creates the app config settings which store the key vault name and Databricks workspace ID, and the app is also configured<\/a> to use the service principal for authentication \/ authorisation. In the auth token input binding<\/a>, found in the function.json file, the correct resources are specified, namely Key Vault and Databricks and the identity is set to clientcredentials, which means that the identity of the function app is used i.e. the service principal.<\/p>\n This means that when the AAD token is generated or when the resource API is invoked, the service principal\u2019s credentials will be used.<\/p>\n Once the app has been published, use the portal development experience to \u201cCode+Test\u201d. Starting with the function to generate the Databricks access token, use the Test functionality, enter a query name \u201cpatsecretname\u201d and value and click Run.<\/p>\n One should receive a 200 OK response and find that a new secret has been stored in Key Vault with the specified name.<\/p>\n Next test the function which creates the Databricks pool, enter a poolsecretname query parameter and ensure that a new pool has been created with the name of the query parameter specified.<\/p>\n This pool will not incur any cost until the instance pool is used by the ADF pipeline – so long as the min_idle_instances parameter in the request payload of the Instance Pool API was set at 0. Any other value and the pool will provision VMs at the minimum threshold specified, incurring standard VM costs.<\/p>\n There will be an associated key vault secret which stores the pool ID, to be used ADF in the linked service configuration.<\/p>\n Note: Whilst a key is required to invoke the functions app, it is still accessible over the public internet. If security is a concern, consider access restrictions<\/a> and whitelisting the IP of the ADF self-hosted integration runtime<\/a> and\/or private site access<\/a> and deploying the integration runtime into a managed vnet<\/a>.<\/p>\n <\/p>\n Using a combination of key vault, parameters and the dynamic contents setting (in the advanced section of the linked service) it is possible to create a more dynamic linked service, into which the configuration details can be \u201cinjected\u201d at runtime.<\/p>\n 1. To begin, grant the managed identity of ADF access to your Azure Key Vault<\/a>.<\/p>\n 2. Then configuring a Key Vault linked service as described in this tutorial<\/a>.<\/span><\/p>\n 3. Next create a new linked service for Azure Databricks, define a name, then scroll down to the advanced section, tick the box to specify dynamic contents in JSON format. Enter the following JSON, substituting the capitalised placeholders with your values which refer to the Databricks Workspace URL<\/a> and the Key Vault linked service created above. Note the new format – adb-..azuredatabricks.net. It is unique per-workspace! This workspace URL could have also been stored in Key Vault also, which is better practice!<\/span><\/p>\n Note that two parameters are created to represent the KV secrets which contain the PAT and the Pool ID. These will be the parameters passed into the pipeline at trigger time.<\/p>\n After the linked service is created it should look as follows:<\/p>\n Note: Personal Access Tokens created by a service principal via the API are not displayed when you are logged into the Workspace UI because they have been generated by a different security principal. They are however visible via token LIST API using the AAD token generated from the service principal created above.<\/p>\n 4. Create another linked service to authenticate to the Azure Function app as shown in the documentation<\/a>.<\/p>\n 5. Next, create a pipeline and add two parameters which will represent the names of the secrets in Key Vault which will contain the access token and pool ID.<\/p>\n 6. Drop two Function activities on to the canvas.<\/p>\n 7. Specify the Function linked service, and using the function name specify each function to be invoked as well as their associated query parameter. For generating the access token use the following expression substituting the function name if necessary:<\/p>\n 8. In the next function activity specify a function name which will create the instance pool, for example:<\/p>\n 9. Connect these two activities and Publish the changes.<\/p>\n 10. Next, create another pipeline and add two parameters which will be passed to the pipeline which execute the Function apps.<\/p>\n 11. On the canvas add an Execute Pipeline activity. Specify the pipeline created above as the invoked pipeline in the Execute Pipeline activity. In the parameters section click on the value section and add the associated pipeline parameters to pass to the invoked pipeline.<\/p>\n 12. Add a Databricks notebook activity and specify the Databricks linked service which requires the Key Vault secrets to retrieve the access token and pool ID at run time.<\/p>\n 13. Add these pipeline parameters to the linked service properties so that they are passed through to the link service at trigger time.<\/p>\n 14. Under the settings tab enter the path of the notebook created in the prerequisites. The path will similar to the following:<\/p>\n 15. Copy and paste the Databricks activity three times and connect all the activities.<\/p>\n 16. Optionally, create another function app and activity which will revoke the access token and delete the instance pool at the end of the pipeline.<\/p>\n 17. Publish the changes and trigger this pipeline, monitoring the results.<\/p>\n <\/p>\n Using only job clusters which spin-up with each Databricks activity, the total time for the same workload is around 18 and half minutes.<\/p>\n Notice how each cluster takes between 4 and 5 minutes per activity.<\/p>\n <\/p>\n Using instance pools the total time dramatically reduces to under 10 minutes.<\/p>\n Notice how despite the first Databricks activity which took the usual 4\u20135 minutes, the remaining activities are around a minute and a half, most of the time reflected is the time taken to initialise the Spark cluster.<\/p>\n Automation and security improvements to the Databricks workspace are being constantly released such as AAD tokens, permissions API, token management API, cluster policies, etc. Instance pools are also another example of these improvements and can make a dramatic improvement to the completion times of your ADF-based Databricks workloads – particularly so when running a series of chained Databricks activities. Managing access to the workspace and provisioning instance pools no longer requires manual intervention when using AAD tokens for workspace automation. Granting just-in-time access to these resources reduces the chance of manual error, promotes better governance and reduces the risk of improper access token practices.<\/p>\n <\/p>\n <\/p>\n Using AAD tokens it is now possible to generate an Azure Databricks personal access token programmatically, and provision an instance pool using the Instance Pools API. The token can be generated and utilised at run-time to provide \u201cjust-in-time\u201d access to the Databricks workspace. Using the same AAD token, an instance pool can also be provisioned and used to run a series of Databricks activities in the same ADF pipeline.<\/p>\n","protected":false},"author":430,"featured_media":36909,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ep_exclude_from_search":false,"_classifai_error":"","_classifai_text_to_speech_error":"","footnotes":""},"categories":[594],"post_tag":[519],"content-type":[],"coauthors":[1197,1476],"class_list":["post-38976","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technetuk","tag-technet-uk"],"yoast_head":"\n![\"An](\"https:\/\/cm-edgetun.pages.dev\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/08\/JiTAD3.png\")
<\/p>\nDemonstration<\/h2>\n
Prerequisites<\/h2>\n
\n
\n
print(\"Workload goes here\")<\/pre>\n
Key Vault Configuration<\/h2>\n
Function App Configuration<\/h2>\n
![\"Azure](\"https:\/\/cm-edgetun.pages.dev\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/08\/JiTAD4.png\")
<\/p>\n![\"Editing](\"https:\/\/cm-edgetun.pages.dev\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/08\/JiTAD5.png\")
<\/p>\n![\"Looking](\"https:\/\/cm-edgetun.pages.dev\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/08\/JiTAD6.png\")
<\/p>\n![\"The](\"https:\/\/cm-edgetun.pages.dev\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/08\/JiTAD7.png\")
<\/p>\n![\"Showing](\"https:\/\/cm-edgetun.pages.dev\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/08\/JiTAD8.png\")
<\/p>\nData Factory Configuration<\/h2>\n
{ \r\n \"properties\": { \r\n \"type\": \"AzureDatabricks\", \r\n \"parameters\": { \r\n \"myadbpatsecretname\": { \r\n \"type\": \"string\" \r\n }, \r\n \"myadbpoolsecretname\": { \r\n \"type\": \"string\" \r\n }\r\n },\r\n \"annotations\": [], \r\n \"typeProperties\": { \r\n \"domain\": \"WORKSPACE URL<\/a>\", \r\n \"accessToken\": { \r\n \"type\": \"AzureKeyVaultSecret\", \r\n \"store\": { \r\n \"referenceName\": \"KEY VAULT LINKED SERVICE NAME\", \r\n \"type\": \"LinkedServiceReference\" \r\n }, \r\n \"secretName\": \"@{linkedService().myadbpatsecretname}\" \r\n }, \r\n \"instancePoolId\": { \r\n \"type\": \"AzureKeyVaultSecret\", \r\n \"store\": { \r\n \"referenceName\": \"KEY VAULT LINKED SERVICE NAME\", \r\n \"type\": \"LinkedServiceReference\" \r\n }, \r\n \"secretName\": \"@{linkedService().myadbpoolsecretname}\" \r\n }, \r\n \"newClusterNodeType\": \"Standard_DS3_v2\", \r\n \"newClusterNumOfWorker\": \"2\", \r\n \"newClusterVersion\": \"6.4.x-scala2.11\" \r\n } \r\n } \r\n}<\/pre>\n![\"The](\"https:\/\/cm-edgetun.pages.dev\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/08\/JiTAD9.png\")
<\/p>\n![\"The](\"https:\/\/cm-edgetun.pages.dev\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/08\/JiTAD10.png\")
<\/p>\n![\"Creating](\"https:\/\/cm-edgetun.pages.dev\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/08\/JiTAD11.png\")
<\/p>\n@concat('createADBPAT?patsecretname=',pipeline().parameters.patsecretname)<\/pre>\n@concat('createADBPool?poolsecretname=',pipeline().parameters.poolsecretname)<\/pre>\n![\"Editing](\"https:\/\/cm-edgetun.pages.dev\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/08\/JiTAD12.png\")
<\/p>\n![\"Creating](\"https:\/\/cm-edgetun.pages.dev\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/08\/JiTAD13.png\")
<\/p>\n![\"Adding](\"https:\/\/cm-edgetun.pages.dev\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/08\/JiTAD14.png\")
<\/p>\n\/Users\/[USERNAME]\/Workload1<\/pre>\n
![\"Connecting](\"https:\/\/cm-edgetun.pages.dev\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/08\/JiTAD15.png\")
<\/p>\nResults<\/h2>\n
With job clusters<\/h3>\n
![\"The](\"https:\/\/cm-edgetun.pages.dev\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/08\/JiTAD16.png\")
<\/p>\n![\"A](\"https:\/\/cm-edgetun.pages.dev\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/08\/JiTAD17.png\")
<\/p>\nWith Instance Pools<\/h2>\n
![\"The](\"https:\/\/cm-edgetun.pages.dev\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/08\/JiTAD18.png\")
<\/p>\n![\"The](\"https:\/\/cm-edgetun.pages.dev\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/08\/JiTAD19.png\")
<\/p>\n![\"A](\"https:\/\/cm-edgetun.pages.dev\/en-us\/industry\/blog\/wp-content\/uploads\/sites\/22\/2020\/08\/JiTAD20.png\")
<\/p>\nConclusion<\/h2>\n
Useful Links<\/h2>\n
\n
Resources for your decision making team<\/h2>\n
\n