Matillion ETL Pipeline and Job Documentation Extraction

This topic briefs about the Matillion ETL pipeline and job documentation extraction prerequisites.

In This Topic:

Introduction
Artifact Extraction
Getting Help

Introduction

LeapLogic Assessment profiles existing inventory, identify complexity, lineage and provides comprehensive recommendations for migration to modern data platform.

Artifact Extraction

LeapLogic requires Matillion pipelines to be exported in the JSON format and job documentation in the HTML format.

Sl. No.	Artifacts needed	Steps	Priority	Description
1	Pipeline	See here	Must have	LeapLogic will consume the extracted pipeline information (JSON files) to perform extensive assessment and generate insights. This will help to create a migration roadmap.
2	Job Documentation	See here	Must have	LeapLogic will consume this (HTML files) to understand the detailed flow of jobs and components along with the properties used in the jobs.

Pipeline Extraction

Please follow the steps given below to export the pipelines from your Matillion environment.

To export jobs, environments, and variables, select the Project menu and then select Export. This will open the Export overlay.

Select the combination of Jobs (in the left panel), Environments (center panel), and/or Variables (right panel) you want to export. You can include all three types in a single export operation.

Note:

We recommend exporting all the database objects in one go for extensive assessment and insights.

Enter a File Name for the exported JSON file and click OK. The exported information is saved in JSON format in a file on the machine the browser window is running on, in whichever default download location is used by your browser and operating system.
Share the exported output. A sample of the exported JSON artifact is as shown below. It contains a orchestration and a transformation job.

Job Documentation Extraction

Job Documentation can be exported in the HTML format. It is helpful in the visualization and understanding of the flow of the ETL pipelines. Below are the steps for exporting the job documentation.

Login to the Matillion instance and select the required project.
Important Note for Bulk Export of All Jobs: For exporting all the jobs in a project in one go, select the root project and then all the folders and sub-folders. Then, follow the procedure from step-3 below to download all the jobs all at once.
Navigate to the Orchestration Job in the project hierarchy for which the Job documentation is required to be generated.
Right click on the Orchestration Job and then select Generate Job Documentation. The Orchestration job documentation encapsulates all the details of the used Transformation Jobs as well.

A dialog box opens. Click OK to download the Job Documentation.

The exported information is saved in the HTML format in a file on the machine the browser window is running on, in whichever default download location is used by your browser and operating system.

Getting Help

Contact LeapLogic technical support at info@leaplogic.io