DBT Artifacts Extraction Prerequisites

This topic briefs about the DBT artifacts extraction prerequisites.

In This Topic:

Introduction
Exporting from Local DBT
Exporting from Cloud DBT

Introduction

LeapLogic Assessment profiles DBT artifacts and identifies existing inventory, calculates complexity, identifies data and process lineage, and provides comprehensive recommendations for migration to modern data platform.

Exporting from Local DBT

Please follow the below steps to start exporting your artifacts from your local DBT.

Generate JSONs

First, you need to generate two JSON files namely catalog.json and manifest.json. These JSON files serve the purpose of gathering essential information from the target system, including database names, schema names, tables, and more. To generate the catalog and manifest JSON files in the designated folder, execute the following command.

$ dbt docs generate – -project-dir /home/impadmin/snf_sample – -target-path /home/impadmin/tmp

where:

– -project-dir: The DBT project for which you wish to generate catalog and manifest JSON files.
– -target-path: The location where you intend to store the generated files.

Copy DBT Models (SQL files)

The DBT project’s ‘models’ folder comprises of all SQL files utilized for data building. Copy the project-specific models to separate directories as needed. For example, see the below command.

cp -r /home/user1/snf_sample/models /home/user/tmp/snf_sample_models

Exporting from Cloud DBT

Please follow the below steps to start exporting your artifacts from your cloud DBT.

Generate JSONs

First, you need to generate two JSON files namely catalog.json and manifest.json. To generate catalog and manifest JSON files in the target folder on cloud-based DBT, execute the following command.

dbt docs generate

For example:

Upon execution of this command, it will generate JSON files and place them in the target folder within the current project directory.

Commit the changes and initiate a pull request. Afterwards, copy the catalog and manifest JSON files from the GitHub or GitLab repository associated with the DBT project.

Copy DBT Models (SQL files)

The ‘models’ folder within the DBT project houses all the SQL files utilized for data building. Perform a ‘git clone’ and then copy the project-specific models to distinct directories.