Configuring Cloud-to-Cloud Transformation

Cloud-to-cloud transformation is a process of transforming all workloads from an existing cloud to a modern cloud. In other words, moving from one cloud to another. This topic provides steps to configure cloud-to-cloud transformation. To do so, follow the below steps.

In Source Type, select the cloud source data store such as Snowflake.

In Input Type, the default input type for the transformation is set to SQL/Procedure.
In Target Type, select the preferred target type, such as Databricks Lakehouse or Databricks Notebook.
In Output Type, the default output type for the transformation is set to Python.

In Input Artifacts, upload the files that you need to transform to the target source.
Click Data Configuration to configure the data.

If the selected target is Databricks Lakehouse, enable Unity Catalog to transform the constraints such as primary key, foreign key, etc., present in the source data to the Databricks-native equivalent. Otherwise, the default Hive meta store will be used, and Hive or Spark tables will be generated without any constraints. In Databricks, there are two types of catalogs:

Hive Meta store: Hive meta store serves as a central repository to create, store, and manage large datasets or tables. However, it does not support constraints like primary key, foreign key, etc.
Unity Catalog: Unity Catalog in Databricks serves as a centralized metadata repository with advanced metadata management features that support constraints including primary key, foreign key, etc. It serves as a comprehensive repository for storing metadata information within Databricks.

In Validation Type, select the validation type such as None or Cluster.

In Source, select the configuration as Live or Offline.
If the selected source configuration is:
1. Live-Upload the data source.
2. Offline-Upload the DDL files. It supports .sql and .zip file formats.

In Target, select the configuration as Live.
Upload the target data source.

Note:

To increase the accuracy, it is recommended to include metadata in the Transformation stage.

In File Format, select the storage type as Delta, ORC, Parquet, etc.
In Mapping as per Changed Data Model, upload files for mapping between source and target tables.

Click Save to save the Transformation stage.
An alert pop-up message appears. This message prompts you to refer to your respective assessment to determine the anticipated quota deduction when converting your scripts. Then click Ok.

Click to provide a preferred pipeline name.

Click to execute the integrated or standalone pipeline. Clicking (Execute) navigates you to the pipeline listing page which shows your pipeline status as Running state. It changes its state to Success when it is completed successfully.

Click on your pipeline card to see reports.

Note:

When you license LeapLogic, it comes with a certain quota limit, for example unit, block, and script. The system alarms the user when the limit exceeds and highlights quota exhaustion in the pipeline reports.