{"id":64268,"date":"2022-08-12T19:38:26","date_gmt":"2022-08-12T18:38:26","guid":{"rendered":"https:\/\/cm-edgetun.pages.dev\/en-gb\/industry\/blog\/?p=64268"},"modified":"2023-02-24T15:59:17","modified_gmt":"2023-02-24T14:59:17","slug":"injecting-lineage-and-attributes-into-microsoft-purview-part-1","status":"publish","type":"post","link":"https:\/\/cm-edgetun.pages.dev\/en-gb\/industry\/blog\/technetuk\/2022\/08\/12\/injecting-lineage-and-attributes-into-microsoft-purview-part-1\/","title":{"rendered":"Injecting lineage and attributes into Microsoft Purview \u2013 Part 1"},"content":{"rendered":"

\"An<\/p>\n

\u201cHow do I easily find the data I\u2019m looking for?\u201d<\/p>\n

\u201cHow do I know the data comes from a reliable source?\u201d<\/p>\n

These are common questions we hear from data consumers, particularly more so as the organisation starts to scale it\u2019s data platform with numerous data sources and data producers. Identifying the right datasets across a vast array of data stores or a heap of catalog search results can be a challenging and time-consuming process. Enriching these consumption-worthy assets with additional business and technical metadata will form part of the solution. This two-part blog describes and demonstrates how to easily add lineage and attributes in order to improve the data consumer\u2019s experience.<\/p>\n

Current capabilities<\/h3>\n

Automatic lineage collection in Microsoft Purview is currently only supported by\u202fcertain ADF activities<\/a>\u202fand\u202fSynapse activities<\/a>. No doubt this will expand over time, however you may wish to capture lineage from other engines such as Synapse Spark or Databricks. Whilst there is a\u202fSpark based lineage collector<\/a>, as well as the\u202fAzure Databricks to Purview Lineage Connector<\/a>\u202fbased on\u202fOpen Lineage<\/a>, you can alternatively inject your own lineage programmatically \u2014 for every transformation or process which creates a new dataset there will need to be some additional code run in order to create the relationship or linkage between inputs and outputs. This solution requires data producers to become diligent about adapting their current and future (ETL) processes to \u201cpublish\u201d this critical metadata to Purview, and this post aims to demonstrate one of the ways this can be achieved.<\/p>\n

Classifications<\/a>\u202fare a good way of describing technical metadata but often consumers need the business understanding\/metadata to compliment this, in which case you may wish to assign attributes or tags to the data. The workflow engine in Microsoft Purview may very well evolve to automate this in the future but for the time being it will have to be a manual applied or a scripted process depending on the business rules\/logic.<\/p>\n

This blog post will show you how to inject both of these metadata elements into your Purview catalog, also known as the Purview\u202fData Map<\/a>. The full source code for this walkthrough can be found in this\u202fGithub repository<\/a>\u202fwhich includes notebooks with markdown comments, but for the remainder of this post we\u2019ll go into a bit more detail. It assumes you have some understanding of Python and how to interact with the\u202fPurview using SDKs\/APIs<\/a>, which includes the prerequisites such as creating a service principal and assigning necessary roles.<\/p>\n

Provenance and Process<\/h3>\n

To inject lineage between datasets which reside in ADLS (data lake) for example, we will use the superb\u202fPyapacheatlas<\/a>\u202fpackage by Will Johnson. Adapted from one of the\u202fsample notebooks<\/a>\u202fin his repository, the notebook used in this blog can be found\u202fhere<\/a>. Ideally perform a git clone on the link to the repo above in order to download both notebooks used in this post.<\/p>\n

To run the code, we\u2019ll use Python in Synapse Spark, as this might be a common point of entry where data producers\/engineers might run their data transformations and will know the inputs and output datasets to their process(es).<\/p>\n

When using Python\u202fdistributions bundled in Apache Spark<\/a>,\u202fone typically needs to attach the libraries either to the cluster or the session, otherwise this happens:<\/p>\n

\"A<\/p>\n

There are a\u202fnumber of ways to manage Python libraries<\/a>\u202fin Synapse Spark however a simple approach is to create a requirements file:<\/p>\n

\"A<\/p>\n

…and upload it to the spark pool:<\/p>\n

\"An<\/p>\n

Once you have opened\/imported the\u202fsample notebook<\/a>\u202finto a Synapse Spark notebook, enter the Purview account and authentication details in cells 2 and 3. In summary, you\u2019ll need to define (and optionally create) both the source and target assets\/entities (using upload_entities) in cell 4, then in the following cell (5) create the relationship (lineage) between these inputs and outputs. If either the inputs or outputs don\u2019t exist at the time, then no relationship can be created. If either exist, such as the input asset, it is not affected by this operation.<\/p>\n

From a timing perspective, typically what may happen is that the input assets already exist in Purview, discovered via a scan, and a Spark job is going to make use of these to produce new assets or target outputs. For existing (scanned) assets, ensuring the fully qualified name is set correctly in the AtlasEntity section of the notebook (as shown below) avoids creating a duplicate asset. The qualified name can be found in the overview tab. Additionally, set the GUID which can be found in the URL when viewing the asset in the Purview UI. Alternatively this information can be obtained via the\u202fdiscovery_query API<\/a>\u202for the\u202fget_entity<\/a>\u202fmethod which would make more sense if it is part of a scripted approach. Enter the entity name and type \u2014 for a full list of type names you can be obtained by using the\u202fGetEntityDefinition API<\/a>.<\/p>\n

Below is an image of the example input and output assets in the notebook. The input json file is going to be transformed by Spark into a set parquet files, which we know will be detected as a resource set. At this stage a scan has not been run on the input location, therefore we are using a new negative GUID to let Purview know this is a new asset.<\/p>\n

\"An<\/p>\n

As mentioned, the upload_entities call ensures that both the inputs and outputs are created in the Data Map so that a relationship can be formed between them. For assets that haven\u2019t been scanned and don\u2019t yet exist in the Data Map, it simply creates a \u201cplaceholder\u201d entity which is visible via the Purview UI. Then we can make a call to define the lineage artifact using the AtlasProcess class passing these input(s) and output(s) \u2014 see image below of cell 5. Make sure to specify a unique process qualified name (process_qn) for each new relationship otherwise you will end up overwriting an existing one with the same name, and existing lineage for the existing process will be lost! This means you may want to think about appending the process_qn with a unique GUID or task\/backlog item number. The process type name should be \u201cprocess\u201d unless you wish to create your own process type using the typedefs API. For reference\u202fhere is an example custom types definition<\/a>. The name of the process can be anything which describes your process. Finally, the upload_entities call creates this relationship, making the lineage visible in the Purview UI.<\/p>\n

\"Example<\/p>\n

Once you populate and run cells 1-5, you should see lineage between the assets. If not,\u202fHit Refresh!<\/a><\/p>\n

\"The<\/p>\n

One can add additional attributes such as a description and owner\/experts using their\u202fobject IDs from AAD. This will enable your data consumers to know who to contact if they have questions about the process. An example is shown below which defines whether the contact is an expert or owner, the AAD object ID and an info field, which can be any text string such an extension number for example.<\/p>\n

process = AtlasProcess(name=\"Synapse Spark - process raw\",\nattributes={\"description\":\"Spark job to transform raw files into standized format\"},\ncontacts={\"Expert\":[{\"id\":\"aaaaaaa-0000-000a-aa00-0a000a0000a0\",\"info\":\"ext 3234\"}],\"Owner\":[{\"id\":\"aaaaaaa-0000-000a-aa00-0a000a0000a0\",\"info\":\"ext 2553\"}]},<\/pre>\n
\"A<\/p>\n
Now you have seen how to inject lineage programmatically, it’s time to think about how this could be included as a repeatable step in your business\u2019 data pipelines. Ideally you may want to convert these cells 1\u20135 into a library so that they can be called in a single function call, passing the required information as parameters. As part of a data pipeline, the engineer will most likely be referencing these (fully qualified) data asset paths for source and target, unless they are using mounts in which case these will need to be translated (ideally programmatically) to the full path otherwise one will end up with duplicate assets in the data map \u2014 (1) the injected asset and (2) the scanned asset.<\/p>\n
In reality you may have a number of source-to-target transformations, and often multiple source datasets are required to produce a single target dataset – this is more representative of a full analytics (ETL) pipeline. You will see an example of this in the last cell of the notebook. To achieve this, simply define multiple input sources (and possibly multiple output targets), and for each new target asset being produced, simply repeat the process of defining inputs\/outputs, AtlasProcess and making the upload_entities call. You may end up with something like this to represent a more realistic pipeline which depicts the standardisation of some data before joining them to produce a final target dataset.<\/p>\n
\"A<\/p>\n
In Part 2, we take a look at how you can inject attributes to enrich the metadata about your assets<\/a>.<\/p>\n
Learn more<\/h3>\n