Configuring On-premises to Cloud Transformation

In the Transformation stage, the legacy codes and the business logic are transformed to the preferred target.

To configure the Transformation stage, follow the below steps:

In Source Type, select the required source data store, such as Netezza, Oracle, SQL Server, Teradata, Vertica, and more.

In Input Type, select the type of input script, such as SQL/BTQ/Procedure or KSH.
In Target Type, select the preferred target type, such as Spark, Snowflake, Amazon Redshift, and so on.
In Output Type, select the output type (such as Python, RSQL) to generate artifacts in that format.

In Input Artifacts, upload the files that you need to transform to the target source.

Click Data Configuration to configure the data.

Note:

If the selected target is Databricks Lakehouse, enable Unity Catalog to transform the constraints such as primary key, foreign key, etc., present in the source data to the Databricks-native equivalent. Otherwise, the default Hive meta store will be used, and Hive or Spark tables will be generated without any constraints. In Databricks, there are two types of catalogs:

Hive Meta Store: Hive meta store serves as a central repository to create, store, and manage large datasets or tables. However, it does not support constraints like primary key, foreign key, etc.
Unity Catalog: Unity Catalog in Databricks serves as a centralized metadata repository with advanced metadata management features that support constraints including primary key, foreign key, etc. It serves as a comprehensive repository for storing metadata information within Databricks.

In Validation Type, select the validation type:

None: Performs no validation.
Cluster: Syntax validation is performed on queries transformed by the LeapLogic Core transformation engine.

In AI Augmentation, select the preferred Transformation and Review models to convert queries that require augmented transformation and perform syntax validation for accuracy and optimal performance. To convert queries using AI Augmentation, you must provide the source and target combinations in the Add New Sources and Targets page (Governance > Intelligence Modernization > Custom Source/ Target > Add New Sources and Targets)

To view the detailed steps for adding source and target combinations in the Add New Sources and Targets page, click here.

Transformation Model: Select the preferred Transformation model to convert queries that are not handled by the default LeapLogic Core transformation engine to the target equivalent. The Transformation Model supports both Open-Source AI models (LeapLogic) and Enterprise AI models (Databricks AI and Amazon Bedrock such as Claude, Nova, etc.) for query transformation. It also identifies query types even if conversion fails.

The Open-Source AI models include:

OpenOrca 7B, GPTQ 4-bit quantized Model (Hugging Face), Llama 8B, 16-bit quantized Model (Ollama), and Llama 8B, 4-bit quantized Model (Ollama): To convert small- to medium-sized SQLs.
Code Llama 34B, GPTQ 4-bit quantized Model (Hugging Face): To convert large-sized SQLs and procedural code.

In Additional Prompt, provide supplementary instructions in addition to system-generated prompts. For example: Ensure the given DDL metadata is applied during conversion and validation for casting, precision, and scale based on the target system.
In Response Fine Tuning, provide examples based on the additional prompt to fine-tune the response to align the generated output with your preferred level of detail. For example, input could be INSERT INTO CUSTOMER (cust_id, cust_name, credit_limit, discount_rate) VALUES (1, ‘JohnDoe’, 12345.6789, 0.123456); and the expected output could be INSERT INTO CUSTOMER (cust_id, cust_name, credit_limit, discount_rate) VALUES (1, ‘JohnDoe’, CAST (12345.6789,numeric(10,2)), 0.123456);

Review Model: Select one or more Review models from to perform syntax validation on the queries transformed by the LeapLogic Core engine and the Transformation Model. The Review Model validates the transformed queries syntactically, excluding any procedural logic, and suggests corrections where required. You can configure multiple Review models. When multiple Review models are configured:

The first Review model validates the queries transformed by the LeapLogic Core engine and the Transformer model. It validates the transformed queries syntactically, excluding any procedural logic. If any queries are identified as incorrectly transformed, it suggests updated queries.
The updated queries are then passed to the next Review model, which validates them and suggests corrections, if required.
This process continues through all configured Review models until the queries are successfully validated, and the optimized queries are generated.

This validation process ensures higher accuracy, better performance, and more efficient transformation.

Notes:

To improve accuracy, it is recommended to include DDL during the transformation stage. This ensures accurate conversion and validation of transformed queries, including:
- Correct handling of precision and scale
- Resolving case-insensitive issues
- Performing column type casting when required
It is recommended to use Enterprise AI models (Amazon Bedrock such as Claude LLM models or Databricks Mosaic AI models) as the Transformation Model. However, you have the flexibility to choose any model as required.
AI Augmentation leverages Databricks Mosaic AI models and Amazon Bedrock models, such as Claude, Nova, and other foundation models to fine-tune queries within the current session. It does not perform cross-customer learning, and all data remains confined to the session. These models operate entirely within LeapLogic’s or the customer’s environment, ensuring IP protection, data sovereignty, and compliance with enterprise security standards.

To access this intelligent modernization feature (AI Augmentation), ensure that your account has the manager and llexpress_executor roles.

To view the detailed steps for assigning manager and llexpress_executor roles to your account, click here.

In Source Configuration, select the source configuration as Live or Offline.
If the selected Source Configuration is:
1. Live: Upload the data source and select the Default Database.
2. Offline: Upload the DDL files. It supports .sql and .zip file formats.

In Target Configuration, select the target configuration as Live.
Upload the target data source.

Note:

To increase the accuracy, it is recommended to include metadata in the Transformation stage.

In File Format, select the storage type as ORC, Parquet, Text File, or Avro
In Mapping as per Changed Data Model, upload files for mapping between source and target tables.

Notes:

While configuring the Transformation stage, if the selected target is either Databricks Lakehouse or Databricks Notebook, the Databricks Recommendations Report field will appear in the Additional Information section. In this field, you can upload the Databricks recommendations (databricksrecommendation.csv) report generated as part of the corresponding assessment, which enables the system to incorporate relevant optimizations into the converted DDL queries.

To obtain databricksrecommendation.csv report follow the steps below:

Go to Reports section in assessment.
Download Insights and Recommendations.
The downloaded archived file contains the databricksrecommendation.csv report. If required, you can update details such as cluster_by_columns.

Upload databricksrecommendation.csv file in the Databricks Recommendations Report field.

If the selected source is Teradata and the target is Kyvos, in the Table-Cube mapping field upload the mapping file. It is used to map between table and cube where table is replaced with <Folder Name> and table alias is replaced with <Cube Name>. The input to the Table-Cube mapping field is the output CSV file generated from the assessment, where you need to add <Folder Name> and <Cube Name> corresponding to the queries manually.

In Triggers, upload the trigger statements for a more accurate conversion.
Click Save to save the Transformation stage.
An alert pop-up message appears. This message prompts you to refer your respective assessment to determine the anticipated quota deduction required when converting your scripts to target. Then click Ok.

Click the Execute icon to execute the integrated or standalone pipeline. Clicking the Execute icon navigates you to the pipeline listing page which shows your pipeline status as Running state. It changes its state to Success when it is completed successfully.

Click on your pipeline card to see reports.

To view the On-premises to Cloud Transformation, visit On-premises to Cloud Transformation Report.

Note:

When you license LeapLogic, it comes with a certain quota limit, for example unit, script, and block. The system alarms the user when the limit exceeds and highlights quota exhaustion in the pipeline reports.