|
Target Type |
Input |
|
Spark |
- Enable Spark-Native: External toggle to fetch input data from an external source such as Oracle, Netezza, Teradata, etc., and process that data in Spark, and then move the processed or output data to an external target. For instance, if the source input file contains data from any external source like Oracle, you need to select Oracle as the Databases to establish the database connection and load the input data. Then data is processed in Spark, and finally the processed or output data gets stored at an external target (Oracle). However, if you select Oracle as the Databases but the source input file contains data from an external source other than Oracle, such as Teradata, then by default, it will run on Spark.
- In Databases, select the database you want to connect to. This establishes the database connection to load data from external sources like Oracle, Teradata, etc. If the database is selected, the converted code will have connection parameters (in the output artifacts) related to the database. If the database is not selected, you need to add the database connection details manually to the parameter file to execute the dataset; otherwise, by default, it executes on Spark.
- In Output Type, the default output type for the transformation is set to Python.
- In Validation Type, select None or Cluster as the mode of validation.
- None: Select this option if you do not want to perform any validation.
- Cluster: Select this option to perform syntax validation.
- In Data Source, upload the corresponding data source. To successfully perform the syntax validation of the transformed queries, it is advisable to ensure that the required tables are created or already present on the target side and secondly all the user-defined functions (UDFs) are registered on the target data source.
- In Default Database, select the source database to act as the default database in the transformed code such as in the converted lookup procedures, etc.
- In Source Data Source, select the data source (DDL) which contains corresponding metadata to ensure accurate query conversion.
|
|
Snowflake |
- In Output Type, select Python, Snowflake Scripting, or DBT as output type format for the generated artifacts.
- In Validation Type, select None or Cluster as the mode of validation.
- None: Select this option if you do not want to perform any validation.
- Cluster: Select this option to perform syntax validation.
- In Data Source, upload the corresponding data source. To successfully perform the syntax validation of the transformed queries, it is advisable to ensure that the required tables are created or already present on the target side and secondly all the user-defined functions (UDFs) are registered on the target data source.
- In Default Database, select the source database to act as the default database in the transformed code such as in the converted lookup procedures, etc.
- In Source Data Source, select the data source (DDL) which contains corresponding metadata to ensure accurate query conversion.
|
|
AWS Glue Studio |
- To choose the required databases, turn on Is JDBC Required toggle and then select the required databases such as Oracle, or SQL Server.
- In Target Database Details, specify database name, schema name, and prefix. The table name displays in prefix_database_tablename format if prefix is provided.
- In AWS Glue Catalog Database, provide the AWS Glue Catalog Database connection details to connect the database and schema.
- In S3 Bucket Base Path, provide the S3 storage repository path where you need to store the source and target files.
- Specify the UDF File Location and UDF Jar Location to define the new UDF location.
- In Parameter File, upload the parameter files to set the key values for the connection.
- In Target Connection Name, provide a descriptive connection name or tag to identify who executed it.
- In Source Data Source, select the data source (DDL) which contains corresponding metadata to ensure accurate query conversion.
|
|
Databricks Notebook |
- In Output Type, select Python 3 or Jupyter as output type format for the generated artifacts.
- In Data Interaction Technique, select your data interaction method. Following are the options:
- Databricks-Native: Select Databricks-Native to fetch, process, and store data in Databricks Lakehouse.
- Databricks: Unity Catalog: Select Databricks: Unity Catalog to access data via Databricks Unity Catalog. In Databricks, the Unity Catalog serves as a metadata repository from which data is fetched, processed, and stored within the catalog.
- Databricks: External: Select this data interaction technique to fetch input data from an external source such as Oracle, Netezza, Teradata, etc., and process that data in Databricks, and then move the processed data or output to an external target. For instance, if the source input file contains data from any external source like Oracle, you need to select Oracle as the Source Database Connection to establish the database connection and load the input data. Then data is processed in Databricks, and finally the processed or output data gets stored at an external target (Oracle). However, if you select Oracle as the Source Database Connection but the source input file contains data from an external source other than Oracle, such as Teradata, then by default, it will run on Databricks.
- If the selected data interaction technique is Databricks: External, you need to specify the source database of your data. In the Source Database Connection, select the database you want to connect to. This establishes the database connection to load data from external sources like Oracle, Teradata, etc. If the database is selected, the converted code will have connection parameters (in the output artifacts) related to the database. If the database is not selected, you need to add the database connection details manually to the parameter file to execute the dataset; otherwise, by default, it executes on Databricks.
- In Validation Type, select None or Cluster as the mode of validation.
- None: Select this option if you do not want to perform any validation.
- Cluster: Select this option to perform syntax validation.
- In Data Source, upload the corresponding data source. To successfully perform the syntax validation of the transformed queries, it is advisable to ensure that the required input tables are created or already present on the target side and secondly all the user-defined functions (UDFs) are registered on the target data source.
- In Default Database, select the source database to act as the default database in the transformed code such as in the converted lookup procedures, etc.
- In Source Data Source, select the data source (DDL) which contains corresponding metadata to ensure accurate query conversion.
|
|
Databricks Lakehouse |
- In Data Interaction Technique, select your data interaction method. Following are the options:
- Databricks-Native: Select Databricks-Native to fetch, process, and store data in Databricks Lakehouse.
- Databricks: Unity Catalog: Select Databricks: Unity Catalog to access data via Databricks Unity Catalog. In Databricks, the Unity Catalog serves as a metadata repository from which data is fetched, processed, and stored within the catalog.
- Databricks: External: Select this data interaction technique to fetch input data from an external source such as Oracle, Netezza, Teradata, etc., and process that data in Databricks, and then move the processed data or output to an external target. For instance, if the source input file contains data from any external source like Oracle, you need to select Oracle as the Source Database Connection to establish the database connection and load the input data. Then data is processed in Databricks, and finally the processed or output data gets stored at an external target (Oracle). However, if you select Oracle as the Source Database Connection but the source input file contains data from an external source other than Oracle, such as Teradata, then by default, it will run on Databricks.
- If the selected data interaction technique is Databricks: External, you need to specify the source database of your data. In the Source Database Connection, select the database you want to connect to. This establishes the database connection to load data from external sources like Oracle, Teradata, etc. If the database is selected, the converted code will have connection parameters (in the output artifacts) related to the database. If the database is not selected, you need to add the database connection details manually to the parameter file to execute the dataset; otherwise, by default, it executes on Databricks.
- In Validation Type, select None or Cluster as the mode of validation.
- None: Select this option if you do not want to perform any validation.
- Cluster: Select this option to perform syntax validation.
- In Data Source, upload the corresponding data source. To successfully perform the syntax validation of the transformed queries, it is advisable to ensure that the required input tables are created or already present on the target side and secondly all the user-defined functions (UDFs) are registered on the target data source.
- In Default Database, select the source database to act as the default database in the transformed code such as in the converted lookup procedures, etc.
- In Source Data Source, select the data source (DDL) which contains corresponding metadata to ensure accurate query conversion.
|
|
Delta Live Tables |
- In Data Interaction Technique, select your data interaction method. Following are the options:
- Databricks-Native: Select Databricks-Native to fetch, process, and store data in Databricks Lakehouse.
- Enable DLT Meta toggle to facilitate the creation of a bronze table within the Databricks Lakehouse. Rather than fetching data directly from the source such as flat files, this feature creates a bronze table (exact replica of the file) within Databricks and helps to refine data during data ingestion. With DLT Meta enabled, flat files are stored as tables within Databricks ensuring efficient data retrieval directly from these tables. This enhancement significantly boosts overall performance.
- In DBFS Base Path, specify the DBFS location where the source flat files and DDL files are stored. This information is required to create the bronze table in Databricks.
- Databricks: Unity Catalog: Select Databricks: Unity Catalog to access data via Databricks Unity Catalog. In Databricks, the Unity Catalog serves as a metadata repository from which data is fetched, processed, and stored within the catalog.
- Databricks: External: Select this data interaction method to fetch input data from an external source such as Oracle, Netezza, Teradata, etc., and process that data in the Databricks, and then move the processed data or output to an external target. For instance, if the source input file contains data from any external source like Oracle, you need to select Oracle as the Source Database Connection to establish the database connection and load the input data. Then this data is processed in Databricks, and finally, the processed or output data gets stored at an external target (Oracle). However, if you select Oracle as the Source Database Connection but the source input file contains data from an external source other than Oracle, such as Teradata, then by default, it will run on Databricks.
- Enable DLT Meta toggle to facilitate the creation of a bronze table within the Databricks Lakehouse. Rather than fetching data directly from the source such as flat files, this feature creates a bronze table (exact replica of the file) within Databricks and helps to refine data during data ingestion. With DLT Meta enabled, flat files are stored as tables within Databricks ensuring efficient data retrieval directly from these tables. This enhancement significantly boosts overall performance.
- If the selected data interaction technique is Databricks: External, you need to specify the source database of your data. In the Source Database Connection, select the database you want to connect to. This establishes the database connection to load data from external sources like Oracle, Teradata, etc. If the database is selected, the converted code will have connection parameters (in the output artifacts) related to the database. If the database is not selected, you need to add the database connection details manually to the parameter file to execute the dataset; otherwise, by default, it executes on Databricks.
- In DBFS Base Path, provide the DBFS base location where the source flat files and DDL files are stored. This information is required to create the bronze table in Databricks.
- In Validation Type, select None or Cluster as the mode of validation.
- None: Select this option if you do not want to perform any validation.
- Cluster: Select this option to perform syntax validation.
- In Data Source, upload the corresponding data source. To successfully perform the syntax validation of the transformed queries, it is advisable to ensure that the required input tables are created or already present on the target side and secondly all the user-defined functions (UDFs) are registered on the target data source.
- In Default Database, select the source database to act as the default database in the transformed code such as in the converted lookup procedures, etc.
- In Dependent Utility Path, provide the DBFS location where the utility files are stored as a wheel binary package. The wheel file contains Python libraries that are required to execute the conversion on the target.
- In Source Data Source, select the data source (DDL) which contains corresponding metadata to ensure accurate query conversion.
|
|
AWS Glue Job |
- In Data Interaction Technique, select your data interaction method. Following are the options:
- Glue: Redshift: Select Glue-Redshift to fetch input data from Amazon Redshift, process it in Glue, and store the processed or output data in Redshift. In this scenario, source data are converted to Redshift whereas temporary or intermediate tables are converted to Spark.
- Glue: Data Catalog: This method accesses data through the data catalog which serves as a metadata repository. Then the data is processed in Glue and the processed or output data gets stored in the data catalog.
- In Storage Format, select the storage format of your data such as Delta or Iceberg.
- Glue: External: Select this data interaction method to fetch input data from an external source such as Oracle, Netezza, Teradata, etc., and process that data in Glue, and then move the processed or output data to an external target. For instance, if the source input file contains data from any external source like Oracle, you need to select Oracle as the Source Database Connection to establish the database connection and load the input data. Then data is processed in Glue, and finally the processed or output data gets stored at the external target (Oracle). However, if you select Oracle as the Source Database Connection but the source input file contains data from an external source other than Oracle, such as Teradata, then by default, it will run on Redshift.
- If the selected data interaction technique is Glue: External, you need to specify the source database of your data. In the Source Database Connection select the database you want to connect to. This establishes the database connection to load data from external sources like Oracle, Teradata, etc. If the database is selected, the converted code will have connection parameters (in the output artifacts) related to the database. If the database is not selected, you need to add the database connection details manually to the parameter file to execute the dataset; otherwise, by default, it executes on Redshift.
- Redshift ETL Orchestrated via Glue: This method accesses, processes, and executes data in Amazon Redshift and uses Glue for orchestration jobs. In this scenario, both source data and intermediate tables are converted to Redshift.
- Glue: Hybrid: This interaction technique can leverage three different data interaction techniques - Glue: Redshift, Glue: Data Catalog, and Glue: External. Depending upon your use case, it can either take input from the data sources such as Redshift, Delta, Iceberg, RDS instance, or an external source; process it in Glue; and move the output (processed) data into the respective data sources. To define the tables and data sources, download the template (CSV file) and specify the tables with their database type etc. For instance, you can select this option if your source tables reside on multiple data sources such as Redshift, Delta, Iceberg, external sources, and RDS instances. It takes input from the specified data sources, processes in Glue, and moves the output (processed) data into the respective data sources.
- In Input File, upload the input file (CSV file) that contains information about tables including their database type and name, to define the tables residing on various data sources.
- In Default Database, select the source database to act as the default database in the transformed code such as in the converted lookup procedures, etc.
- In Source Data Source, select the data source (DDL) which contains corresponding metadata to ensure accurate query conversion.
|
|
Matillion ETL |
- In Output Type, the default output type for the transformation is set to JSON.
- In Data Interaction Technique, select your data interaction method. Following are the options:
- Snowflake - Native: Select Snowflake - Native to fetch, process, and store data in Snowflake.
- Snowflake: External: Select this data interaction technique to fetch input data from an external source such as Oracle, Netezza, Teradata, etc., and process that data in Snowflake, and then move the processed data or output to an external target. For instance, if the source input file contains data from any external source like Oracle, you need to select Oracle as the Source Database Connection to establish the database connection and load the input data. Then data is processed in Snowflake, and finally the processed or output data gets stored at an external target (Oracle). However, if you select Oracle as the Source Database Connection but the source input file contains data from an external source other than Oracle, such as Teradata, then by default, it will run on Snowflake.
- If the selected data interaction technique is Snowflake: External, you need to specify the source database of your data. In the Source Database Connection, select the database you want to connect to. This establishes the database connection to load data from external sources like Oracle, Teradata, etc. If the database is selected, the converted code will have connection parameters (in the output artifacts) related to the database. If the database is not selected, you need to add the database connection details manually to the parameter file to execute the dataset; otherwise, by default, it executes on Snowflake.
- In Default Database, select the source database to act as the default database in the transformed code such as in the converted lookup procedures, etc.
- In Source Data Source, select the data source (DDL) which contains corresponding metadata to ensure accurate query conversion.
|
|
Redshift ELT |
- Enable Redshift: External toggle to fetch input data from an external source such as Oracle, Netezza, Teradata, etc., and process that data in Redshift, and then move the processed or output data to an external target. For instance, if the source input file contains data from any external source like Oracle, you need to select Oracle as the Databases to establish the database connection and load the input data. Then data is processed in Spark, and finally the processed or output data gets stored at an external target (Oracle). However, if you select Oracle as the Databases but the source input file contains data from an external source other than Oracle, such as Teradata, then by default, it will run on Redshift.
- In Databases, select the database to define the database connection to load the data from external sources such as Oracle, Teradata, etc. If the database is selected, the converted code will have connection parameters (in the output artifacts) related to the database. If the database is not selected, you need to add the database connection details manually to the parameter file to execute the dataset; otherwise, by default, it executes on Redshift.
- In Output Type, the default output type for the transformation is set to Python.
- In Database Name, provide the target database name to which you need to store the transformed data.
- In S3 Bucket Base Path, provide the S3 storage repository path where you need to store the source and target files.
- In Parameter File, upload the parameter files to set the key values for the connection.
- In Target Connection Name, provide a descriptive connection name or tag to identify who executed it.
- In Source Data Source, select the data source (DDL) which contains corresponding metadata to ensure accurate query conversion.
|
|
AWS Glue Notebook |
- In Data Interaction Technique, select your data interaction method. Following are the options:
- Glue-Redshift: Select Glue-Redshift to fetch input from Amazon Redshift, process it in Glue, and store the processed data or output in Redshift. In this scenario, source data are converted to Redshift whereas temporary or intermediate tables are converted to Spark.
- Glue: Data Catalog: This method accesses data through the data catalog which serves as a metadata repository. Then the data is processed within Glue and the processed or output data gets stored in the data catalog.
- In Storage Format, select the storage format of your data such as Delta or Iceberg.
- Glue: External: Select this data interaction method to fetch input data from an external source such as Oracle, Netezza, Teradata, etc., and process that data in the Glue, and then move the processed or output data to an external target. For instance, if the source input file contains data from any external source like Oracle, you need to select Oracle as the Source Database Connection to establish the database connection and load the input data. Then data is processed in Glue, and finally, the processed or output data gets stored at an external target (Oracle). However, if you select Oracle as the Source Database Connection but the source input file contains data from an external source other than Oracle, such as Teradata, then by default, it will run on Redshift.
- If the selected data interaction technique is Glue: External, you need to specify the source database of your data. In the Source Database Connection select the database you want to connect to. This establishes the database connection to load data from external sources like Oracle, Teradata, etc. If the database is selected, the converted code will have connection parameters (in the output artifacts) related to the database. If the database is not selected, you need to add the database connection details manually to the parameter file to execute the dataset; otherwise, by default, it executes on Redshift.
- Redshift ETL Orchestration via Glue: This method accesses, processes, and executes data in Amazon Redshift and uses Glue for orchestration jobs. In this scenario, both source data and intermediate tables are converted to Redshift.
- In Default Database, select the source database to act as the default database in the transformed code such as in the converted lookup procedures, etc.
- In Property file path, provide the S3 repo path where the property files are stored.
- In Dependent Utility Path, provide the S3 repo path where the utility files are stored as a wheel binary package.
- In Source Data Source, select the data source (DDL) which contains corresponding metadata to ensure accurate query conversion.
|
|
Data Build Tool |
- In Data Interaction Technique, select your data interaction method. Following are the options:
- Snowflake - Native: Select Snowflake - Native to fetch, process, and store data in Snowflake.
- Snowflake: External: Select this data interaction technique to fetch input data from an external source such as Oracle, Netezza, Teradata, etc., and process that data in Snowflake, and then move the processed data or output to an external target. For instance, if the source input file contains data from any external source like Oracle, you need to select Oracle as the Source Database Connection to establish the database connection and load the input data. Then data is processed in Snowflake, and finally the processed or output data gets stored at an external target (Oracle). However, if you select Oracle as the Source Database Connection but the source input file contains data from an external source other than Oracle, such as Teradata, then by default, it will run on Snowflake.
- If the selected data interaction technique is Snowflake: External, you need to specify the source database of your data. In the Source Database Connection, select the database you want to connect to. This establishes the database connection to load data from external sources like Oracle, Teradata, etc. If the database is selected, the converted code will have connection parameters (in the output artifacts) related to the database. If the database is not selected, you need to add the database connection details manually to the parameter file to execute the dataset; otherwise, by default, it executes on Snowflake.
- In Default Database, select the source database to act as the default database in the transformed code such as in the converted lookup procedures, etc.
- In Source Data Source, select the data source (DDL) which contains corresponding metadata to ensure accurate query conversion.
|
|
Google BigQuery |
- In Orchestration Technique, select your orchestration method. Following are the options:
- Google Cloud Composer: Select this option to generate Google Cloud Composer equivalent artifacts through which you can create, monitor, and manage the workflows defined in the DAG (Directed Acyclic Graph).
- Python: Select this option to generate Python artifacts.
- In Data Interaction Technique, select your data interaction method from the following options:
- Google BigQuery - Native: Select this option to access the input data, process it, and store the output data in Google BigQuery.
- Google BigQuery: External: Select this option to fetch input data from an external source such as Oracle, Netezza, Teradata, etc., and process that data in Google BigQuery, and then move the processed or output data to an external target. For instance, if the source input file contains data from any external source like Oracle, you need to select Oracle as the source database to establish the database connection and load the input data. Then data is processed in Google BigQuery, and finally the processed or output data gets stored at the external target (Oracle). However, if you select Oracle as the source database connection but the source input file contains data from an external source other than Oracle, such as Teradata, then by default, it will run on Google BigQuery.
- If the selected data interaction technique is Google BigQuery: External, you need to specify the source database of your data. In the Source Database Connection, select the database you want to connect to. This establishes the database connection to load data from external sources like Oracle, Teradata, etc. If the database is selected, the converted code will have connection parameters (in the output artifacts) related to the database. If the database is not selected, you need to add the database connection details manually to the parameter file to execute the dataset; otherwise, by default, it executes on Google BigQuery.
- In Validation Type, select None or Cluster as the mode of validation.
- None: Select this option if you do not want to perform any validation.
- Cluster: Select this option to perform syntax validation.
- In Data Source, upload the corresponding data source. To successfully perform the syntax validation of the transformed queries, it is advisable to ensure that the required input tables are created or already present on the target side and secondly all the user-defined functions (UDFs) are registered on the target data source.
- In GCS Base Path, specify the GCS base path where external files are stored.
- In Default Database, select the source database to act as the default database in the transformed code such as in the converted lookup procedures, etc.
- In Source Data Source, select the data source (DDL) which contains corresponding metadata to ensure accurate query conversion.
|
Leaplogic