Configuring Meta Diff Stage

The Meta Diff stage compares the columns in the source and target datasets. The input for this transformation is entities that require schema comparison.

In This Topic:

Overview
Transform
Output

Overview

In this section, you can customize the name for the Meta Diff stage and can give a suitable description that describes the purpose and scope of the stage. By default, Meta Diff is provided in the Name field.

Transform

In this stage, the source and target data columns are compared to confirm that the data is valid. The entities from the source dataset (Data Source 1) are compared with the entities in the target dataset (Data Source 2) and highlight the post comparison results in a report after the execution of the pipeline.

To configure the Meta Diff stage, follow the below steps:

To upload the source data for comparison, follow the below steps:
1. Click Data Source (DS)1.
2. Choose repository.
3. Select data source.
4. Click to save the source data source (Data Source1).
To upload the target data source for comparison, follow the below steps:
1. Click Data Source (DS)2.
2. Choose repository.
3. Select data source.
4. Click to save the target data source (Data Source2).
In DS 1 Entities, select:
1. Generic-To compare all column across all entities.
2. Entity Specific-To compare all column from the selected entities. To choose the entities, select the entities from the Entities-Specific list.
Click Save to save the Meta Diff stage. When a Meta Diff stage is successfully saved, the system displays an alerting snackbar pop-up to notify the success message.

Click to execute the pipeline. Clicking (Execute) navigates you to the pipeline listing page which shows your pipeline status as Running state. It changes its state to Success when it is completed successfully.

Click on your pipeline card to see reports.

To view the Meta Diff Stage report, visit Meta Diff Report.

Output

In this section, you can configure the output of this stage for navigation to a further stage in case of any error. By default, the Output configuration is set to Error whenever the system encounters an error, but that can be configured to Continue, Stop, or Pause as required.

Next:

Configuring Data Load