Skip to the content
LeaplogicLeaplogic
  • Home
  • About Us
  • Contact
SIGN IN
  • Home
  • About Us
  • Contact

  • Getting Started
    • Before You Begin
    • Creating an Account
    • Logging into LeapLogic
    • Reset Password
    • Quick Tour of the Web Interface
    • LeapLogic in 15 minutes
      • Prerequisites
      • Step 1. Log into LeapLogic
      • Step 2. Create Assessment and Get Insights
      • Step 3. Create Transformation Pipeline and See Results
      • Step 4. Edit or Optimize the Transformed Code
      • Step 5: Complete the Transformation Lifecycle
  • Introduction to LeapLogic
    • Overview
    • High Level Architecture
    • Supported Legacy and Cloud Platforms
    • Key Features
  • LeapLogic Deployment Architecture & Prerequisites
  • Workload Assessment
    • Overview
    • Value Proposition
    • Creating Assessment
      • Prerequisites
      • Step 1. Provide Primary Inputs
        • Automation Coverage
      • Step 2. Add the Additional Inputs
        • Table Stat Extraction Steps
          • Teradata
          • Oracle
          • Netezza
      • Step 3. Update the Source Configuration
      • Step 4. Configure the Recommendation Settings
    • Assessment Listing
    • Understanding Insights and Recommendations
      • Volumetric Info
      • EDW
        • Oracle
          • Highlights
          • Analysis
          • Optimization
          • Lineage
          • Recommendations
          • Downloadable Reports
        • Vertica
          • Highlights
          • Analysis
          • Optimization
          • Lineage
          • Recommendations
          • Downloadable Reports
        • Snowflake
          • Highlights
          • Analysis
          • Optimization
          • Lineage
          • Recommendations
          • Downloadable Reports
        • Azure Synapse
          • Highlights
          • Analysis
          • Optimization
          • Lineage
          • Recommendations
          • Downloadable Reports
        • SQL Server
          • Highlights
          • Analysis
          • Optimization
          • Lineage
          • Recommendations
          • Downloadable Reports
        • Teradata
          • Highlights
          • Analysis
          • Optimization
          • Lineage
          • Recommendations
          • Downloadable Reports
        • Netezza
          • Highlights
          • Analysis
          • Optimization
          • Lineage
          • Recommendations
          • Downloadable Reports
        • Google Big Query
          • Highlights
          • Analysis
          • Optimization
          • Lineage
          • Recommendations
          • Downloadable Reports
        • Redshift
          • Highlights
          • Analysis
          • Optimization
          • Lineage
          • Recommendations
          • Downloadable Reports
        • PostgreSQL
          • Highlights
          • Analysis
          • Optimization
          • Lineage
          • Recommendations
          • Downloadable Reports
        • Duck DB
          • Highlights
          • Analysis
          • Optimization
          • Lineage
          • Recommendations
          • Downloadable Reports
        • ClickHouse
          • Highlights
          • Analysis
          • Optimization
          • Lineage
          • Recommendations
          • Downloadable Reports
        • Exasol
          • Highlights
          • Analysis
          • Optimization
          • Lineage
          • Recommendations
          • Downloadable Reports
        • DB2
          • Highlights
          • Analysis
          • Optimization
          • Recommendations
          • Lineage
          • Downloadable Reports
      • ETL
        • Informatica
          • Highlights
          • Analysis
          • Lineage
          • Downloadable Reports
        • Ab Initio
          • Highlights
          • Analysis
          • Lineage
          • Downloadable Reports
        • DataStage
          • Highlights
          • Analysis
          • Lineage
          • Downloadable Reports
        • Talend
          • Highlights
          • Analysis
          • Lineage
          • Downloadable Reports
        • SSIS
          • Highlights
          • Analysis
          • Lineage
          • Downloadable Reports
        • Informatica BDM
          • Highlights
          • Analysis
          • Lineage
          • Downloadable Reports
        • Oracle Data Integrator
          • Highlights
          • Analysis
          • Downloadable Reports
        • Pentaho
          • Highlights
          • Analysis
          • Downloadable Reports
        • Azure Data Factory
          • ARM Template
          • Highlights
          • Analysis
          • Downloadable Reports
        • Matillion
          • Highlights
          • Analysis
          • Downloadable Reports
        • SnapLogic
          • Highlights
          • Analysis
          • Downloadable Reports
      • Orchestration
        • AutoSys
          • Highlights
          • Analysis
          • Downloadable Reports
        • Control-M
          • Highlights
          • Analysis
          • Lineage
          • Downloadable Reports
        • SQL Server
          • Highlights
          • Analysis
      • BI
        • OBIEE
          • Highlights
          • Analysis
          • Lineage
          • Downloadable Reports
        • Tableau
          • Highlights
          • Analysis
          • Lineage
          • Downloadable Reports
        • IBM Cognos
          • Highlights
          • Analysis
          • Downloadable Reports
        • MicroStrategy
          • Highlights
          • Analysis
          • Lineage
          • Downloadable Reports
        • Power BI
          • Highlights
          • Analysis
          • Lineage
          • Downloadable Reports
        • SSRS
          • Highlights
          • Analysis
          • Downloadable Reports
        • SAP BO
          • Highlights
          • Analysis
          • Lineage
          • Downloadable Reports
        • WebFOCUS
          • Highlights
          • Analysis
          • Downloadable Reports
      • Analytics
        • SAS
          • Highlight
          • Analysis
          • Lineage
          • Downloadable Reports
        • Alteryx
          • Highlights
          • Analysis
          • Lineage
          • Downloadable Reports
      • Shell
        • Highlights
        • Analysis
        • Lineage
        • Downloadable Reports
      • Integrated Assessment (EDW, ETL, Orchestration, BI)
        • Highlights
        • Analysis
        • Optimization
        • Lineage
        • Recommendations
    • Managing Assessment Reports
      • Downloading Report
      • Input Report Utility
      • View Configuration
    • Lineage - Dependency Analysis
    • Column-Level Lineage
      • Manage Lineage
    • Complexity Calculation Logic
    • Key Benefits
    • Ad hoc Query
  • Script/ Query Log/ Code Extraction Prerequisites
    • Cloud
      • Azure Data Factory
      • Snowflake
      • Azure Synapse
      • Google BigQuery
      • Redshift
      • Azure SQL Database Hyperscale
      • Aurora PostgreSQL
    • EDW
      • Oracle
      • Netezza
      • Teradata
      • Vertica
      • SQL Server
      • Db2
      • MySQL
      • PostgreSQL
    • ETL
      • DataStage
      • Informatica
      • SSIS
      • Talend
      • ODI
      • IICS
      • DBT
      • Pentaho
      • Matillion
      • SnapLogic
      • Ab Initio
      • SAP BODS
      • TAC
      • WebFOCUS
    • BI
      • IBM Cognos
      • OBIEE
      • Tableau
      • Metabase
      • MicroStrategy
      • PowerBI
      • LeapLogic Utility for SAP BO
      • SAP BO Universe and Web Intelligence
      • SSRS
    • Analytics
      • SAS
      • Alteryx
    • Orchestration
      • AutoSys
      • Control-M
      • SQL Server
    • Mainframe
  • Metadata Management
    • Overview
    • Introduction to Data Catalog
      • Managing Data Catalog
        • Building Data Catalog
        • Insights to Data Catalog
        • Managing the Repository and Data Source
      • Creating Repository (Repo)
      • Creating Data Source
    • Tag Management
    • Key benefits
  • Batch Processing using Pipeline
    • Introduction
    • Designing Pipeline
      • How to create a pipeline
        • Configuring Migration Stage
          • Schema Optimization
        • Configuring Transformation Stage
          • On-premises to Cloud
          • Cloud-to-Cloud
            • Assigning Roles
            • Intelligent Modernization
          • LeapFusion
        • Configuring Validation Stage
          • Data Validation
            • Table
            • File
            • File and Table
            • Cell-by-cell validation
          • Query Validation
            • Query Validation (When Data is Available)
            • Query Validation (When Data is Not Available)
          • Schema Validation
        • Configuring Execution Stage
        • Configuring ETL Conversion Stage
          • Ab Initio
          • Informatica
          • Informatica BDM
          • Matillion
          • DataStage
          • SSIS
          • IICS
          • Talend
          • Oracle Data Integrator
          • Pentaho
          • SnapLogic
        • Configuring Mainframe Conversion Stage
          • Cobol
          • JCL
        • Configuring Orchestration Stage
          • AutoSys
          • Control-M
        • Configuring BI Conversion Stage
          • OBIEE to Power BI
          • OBIEE to AWS QuickSight
          • Tableau to Amazon QuickSight
          • Tableau to Power BI
          • Tableau to Superset
          • Tableau to Looker
          • IBM Cognos to Power BI
        • Configuring Analytics Conversion Stage
          • SAS
          • Alteryx
        • Configuring Script Conversion Stage
    • Key Features
      • How to schedule a pipeline
      • Configuring Parameters
  • Pipeline Reports
    • Overview of Pipeline Report
    • Pipeline Listing
    • Reports and Insights
      • Migration
      • Transformation
        • On-premises to Cloud
        • Cloud-to-Cloud
      • Validation
        • Data
          • File
          • Table
          • File and Table
        • Query
          • Query Validation Report (When Data is Available)
          • Query Validation Report (When Data is not Available)
        • Schema
      • Execution
      • ETL
        • Ab Initio
        • Informatica
        • Informatica BDM
        • Matillion
        • DataStage
        • SSIS
        • IICS
        • Talend
        • Oracle Data Integrator
        • Pentaho
        • SnapLogic
      • Mainframe
        • Cobol
        • JCL
      • Orchestration
        • AutoSys
        • Control-M
      • BI
        • OBIEE to Power BI
        • OBIEE to Amazon QuickSight
        • Tableau to Amazon QuickSight
        • Tableau to Power BI
        • Tableau to Superset
        • Tableau to Looker
        • IBM Cognos to Power BI
      • Analytics
        • SAS
        • Alteryx
      • Shell Script
      • Common Model
    • Automation Level Indicator
      • ETL
        • Informatica
        • Matillion
        • DataStage
        • Informatica BDM
        • SnapLogic
        • IICS
        • Ab Initio
        • SSIS
        • Talend
        • Pentaho
      • Orchestration
        • AutoSys
        • Control-M
      • EDW
      • Analytics
        • SAS
        • Alteryx
      • BI
      • Shell Script
    • Error Specifications & Troubleshooting
  • SQL Transformation
    • Overview
    • Creating and Executing the Online Notebook
      • How to Create and Execute the Notebook
      • Supported Features
    • Configuring the Notebook
      • Transformation
      • Unit Level Validation
      • Script Level Validation
    • Notebook Listing
  • Operationalization
    • Overview
      • Basic
      • Advanced
      • Cron Expression
    • Parallel Run Pipeline Listing
  • Transformation Source
    • Introduction
    • Creating Transformation Source Type
  • Governance
    • Summary of Governance - Roles and Permissions
    • User Creation
      • Creating a new User Account
    • Adding Roles and permissions
      • How to add Roles and Permissions to a new user?
    • Adding Group Accounts
    • Manage Quota
    • Product Usage Metrics
  • License
    • Workload Assessment
    • EDW Conversion
    • ETL Conversion
    • BI Conversion
    • Orchestration Conversion
    • Data Migration
    • Data Validation
  • LeapLogic Desktop Version
    • Overview
    • Registration and Installation
    • Getting Started
    • Creating Assessment
      • ETL
      • DML
      • Procedure
      • Analytics
      • Hadoop
    • Reports and Insights
      • Downloadable Reports
      • Reports for Estimation
    • Logging and Troubleshooting
    • Sample Scripts
    • Desktop vs. Web Version
    • Getting Help
  • LeapLogic (Version 5.0) Deployment
    • System Requirements
    • Prerequisites
    • Deployment
      • Extracting Package
      • Placing License Key
      • Executing Deployment Script
      • Accessing LeapLogic
    • Uploading License
    • Appendix
    • Getting Help
  • Removed Features
    • Configuring File Validation Stage
    • Variable Extractor Stage
      • Variable Extractor Report
    • Configuring Meta Diff Stage
      • Meta Diff
    • Configuring Data Load Stage
      • Data Load
    • Configuring Multi Algo Stage
  • FAQs
  • Tutorial Videos
  • Notice
Home   »  Batch Processing using Pipeline  »  Introduction

Introduction

Pipeline is a key module in LeapLogic that automates and manages multiple stages of workload modernization. It enables you to configure and execute stages such as migration, transformation, validation, and execution.
The absence of automated business logic transformation often results in fragile rewrites, inconsistent logic, and sunk costs—undermining the momentum built during assessment. The impact can be significant, leading to challenges such as:




LeapLogic Transformation Capabilities

LeapLogic automates end-to-end business logic and workload transformation, converting legacy data warehouse, ETL, BI, analytics, and other workloads into optimized, cloud-native equivalents. Its intelligent, AI-driven engine ensures accuracy, functional integrity, and performance at scale. LeapLogic’s pioneering Workload Transformation is redefining the art of migration and unlocking the full potential of the cloud.

Note:

IntelliParse™, SemantiSync™, etc. are conceptual capability descriptors used for marketing communication; actual feature names may vary within the product.

The table below summarizes the key features and capabilities of LeapLogic’s transformation process, highlighting its comprehensive coverage across logic modernization, automation, validation, security, and so on.

Feature Description
Logic transformation spectrum (Breadth) Legacy platforms and stacks: Data warehouse, ETL, Orchestration, BI Reports, Analytics (SAS), Mainframe, Big data workloads – Hadoop ecosystem workloads, Cloud-native workloads (cloud to cloud migration)
Logic transformation spectrum (Depth) End-to-end business logic including DML, DDL, Procedural code, Shell scripts, ETL scripts, BI Reports, Macros, Scheduler scripts, Orchestrator scripts, Analytics scripts, Cobol/JCL, User Defined Functions, Views, Hive, Spark, Oozie, Sqoop, custom scripts, cloud SQLs, cloud procedures, etc.
Myriad architectural choices Migrates to single-cloud, multi-cloud, hybrid-cloud, serverless architecture/fully managed services. Data estate modernization as per the target native architecture’s leading practices and established standards.
Data interaction method/technique Intelligent transformation as per the preferred data interaction method such as end-to-end native processing, or from and into an external source, or orchestration via a different native service, and more…
Automation coverage Up to 90% in the first iteration with up to 100% in subsequent iterations
Cloud mindset and strategy Automatically assesses, recommends, customizes, and optimizes FOR THE CLOUD as per the native cloud’s best practices
Cloud-to-cloud modernization for interoperability Enables cloud-to-cloud modernization from one cloud service provider or hyperscaler to another for maximum interoperability
Changed data model based intelligent conversion LeapLogic natively supports intelligent, change-aware, and fully automated translation of legacy workloads to modern cloud-native architectures. Its engine dynamically interprets evolving data models and seamlessly rewrites or redesigns transformations to align with target paradigms such as slowly changing dimensions (SCD), data mesh, data fabric, and Lakehouse architectures. By embedding this capability within the translation process itself, LeapLogic eliminates manual intervention, ensures architectural consistency, and accelerates modernization while preserving business logic integrity.
Meet/Beat SLAs – inherent (automated) optimization and performance tuning Automated performance tuning and optimization at schema, code, and orchestration level. Several built-in code-level optimizations eradicating technical debt and transforming as per the cloud’s best practices. Meets and even beats business SLAs consistently in all engagements along with best architecture enforcement.
Custom use cases and discrete enterprise scenarios Customized use cases like rapid extensibility for newer patterns, integration with third-party tools, architectural blueprint as per choice of target services, data ingestion strategy as per org’s security needs, etc. LeapLogic’s USP is logic migration (with inherent optimizations) to industry-standard best architecture, and at the same time, customizing for each customer's unique architecture and design needs.
Validation & certification Automated, pattern-driven data quality (DQ) checks and anomaly detection, auto-generates reconciliation scripts, SQL/query and data-level validation, cell-level reports, system integration testing, remediation workflows using automated validation pipelines.
Code Packaging, orchestration, and execution Code packaging as production-ready jobs along with orchestration logic for seamless execution in the new target environment.
Optimized execution meeting desired SLAs LeapLogic’s advanced execution framework enabling parallel execution of the transformed workloads, restartability, execution lineage, ultimately, meeting business SLAs and ensuring a controlled price-capacity-performance ratio in the new modernized environment.
Security and governance LeapLogic enforces end-to-end data security—including masking, tokenization, and related controls—both at rest and in transit during migration. Since it operates as a standalone deployment within the client’s environment, all existing enterprise and target cloud security configurations are applied without exception. For governance, LeapLogic provides native integration with data catalogs such as Unity Catalog and AWS Glue, enabling seamless policy enforcement and lineage tracking.
Operationalization, cut-over and go-live Productionizes all transformed workloads enabling a smooth cut-over and go-live including parallel runs, snapshotting and backup integration, automated rollbacks, audit logs and change tracking




Stages in Pipeline

Pipeline encompasses multiple stages designed to facilitate data and schema migration, code transformation, validation, and execution of workloads in batches.

These stages include:

  • Migration: Converts schemas to target-equivalent formats and migrates data to target tables.
  • Transformation: Transforms legacy EDW scripts to target-native equivalent code.
  • ETL Conversion: Converts ETL scripts such as Informatica, DataStage, Ab Initio, etc., to target compatible workflows.
  • BI Conversion: Transforms legacy BI scripts into modern target platform equivalent.
  • Analytics Conversion: Transforms legacy analytics workloads such as SAS into modern target platform equivalent.
  • Mainframe Conversion: Transforms Mainframe scripts such as Cobol and JCL to the target cloud platform.
  • Script Conversion: Transforms Shell statements and blocks to the target equivalent Python artifacts.
  • Orchestration: Transforms legacy scheduler scripts into modern scheduler tools.
  • Validation: Validates the migrated data and the transformed queries.
  • Execution: Executes the transformed artifacts on the target environment.

You can configure a pipeline in an integrated fashion or also as individual operations as required. It works in both ways. The integrated one eases the configuration part as the subsequent stages in the pipeline automatically consume the input from the previous stage.

This extremely easy-to-use interface enables you to perform specific tasks through individual transformations. Just drag and drop the required transformation, configure it with the required workloads, execute the pipeline, and see the reports.


To learn more, contact our support team or write to: info@leaplogic.io

Copyright © 2025 Impetus Technologies Inc. All Rights Reserved

  • Terms of Use
  • Privacy Policy
  • License Agreement
To the top ↑ Up ↑