DataStage Assessment Report
This topic contains information about the DataStage assessment report. The assessment assesses workloads and produces in-depth insights that help to plan the migration. DataStage assessment uses XML or DSX files as input.
In This Topic:
Highlights
The highlights section gives you a high-level overview of your assessment summary of the analytics performed on the selected workloads. It includes a graphical depiction of the complexity of files as well as a summary of the files used.
Summary
This section highlights the input DataStage files that were analyzed throughout the various jobs. Here, you can see the number of files, entities, jobs, components, as well as analyzed percentage of the workloads.
Complexity
This section provides a summarized graphical representation of the complexity of the DataStage files that help in making different decisions including migration planning and estimating budget.
Analysis
This topic provides a detailed examination of source files, jobs, and artifacts.
Source Analysis
This section provides a comprehensive report on the source files. It includes information about the total number of files, jobs, components, complexity, percentage of analyzed files, and queries associated with each file.
- Files: Displays name of the file.
- Components: Displays the number of components in each file. DataStage accommodates multiple components like Jobs.
- Analyzed (%): Provides percentage of analyzed queries in the file.
- Jobs: Displays the number of jobs. Jobs contain instructions to perform a set of activities. Click the dropdown arrow next to the job count to view the associated job types along with their counts. The job types include:
- Parallel: Jobs run in parallel.
- Sequential: Jobs run in sequence.
- Others: Jobs that are not parallel or sequential.
- Queries: Provides the total number of queries in the file.
- Routines: Displays the number of routines in each file. It is a custom script or pre-defined collection of functions to create, update or view jobs.
Entities
This section displays a detailed analysis of the entities. It includes information about the job types, jobs, and the associated source files.
- Table Name: Name of the table.
- Type: Displays the type of job such as parallel, sequential, and others.
- Parallel: Jobs run in parallel.
- Sequential: Jobs run in sequence.
- Others: Jobs that are not parallel or sequential.
- Job Name: Displays the associated jobs of each table.
- File Name: Displays the associated source file of each table.
Jobs
This section provides a complete analysis of the jobs. Jobs are the orchestration scripts (such as AutoSys or Control-M) that run in a certain order to perform a set of activities. It provides information regarding associated files that can run in parallel or sequential to perform the task.
Parallel Jobs
This section provides details of jobs that run in parallel.
- Jobs Name: Displays name of the parallel job.
- Associated File: Displays the files associated with the jobs.
Sequential Jobs
This section provides details of jobs that run in sequence.
- Jobs Name: Displays name of the sequence job.
- Associated File: Displays the files associated with the jobs.
- Dependent Jobs: Provides the jobs that are associated with the jobs in the Jobs Name field.
Others
This section provides details of jobs other than parallel and sequential jobs.
- Jobs Name: Displays the name of the job.
- Associated File: Displays the files associated with the jobs.
- Dependent Jobs: Provides the jobs that are associated with the jobs in the Jobs Name field.
Artifacts
This page gives details about artifacts-collections of related server data. It provides a list of missing artifacts, artifacts that appear additionally, or could not be parsed completely due to some error.
Missing Artifacts
This section provides the details of all the missing artifacts. Additionally, it categorizes the missing artifacts into files and entities that are missing.
- Artifact Name: Displays name of the artifact.
- Type: Provides the type of the artifacts such as file, table, etc.
- Linkage: Provides the linked or associated file name.
Additional Artifacts
This section provides the details of all the artifacts that appear additionally. It also categorizes the additional artifacts into files and entities that appeared additionally.
- Artifact Name: Displays name of the artifact.
- Type: Provides the type of the artifacts such as file, table, etc.
- Linkage: Provides the linked or associated file name.
Unparsed Artifacts
This section provides the details of all the artifacts that could not be parsed completely due to some error.
- File Name: Displays name of the artifact.
- Message: Displays the message.
Lineage
End-to-end data and process lineage identify the complete dependency structure through interactive and drill-down options to the last level.
Typically, even within one line of business, multiple data sources, entry points, ETL tools, and orchestration mechanisms exist. Decoding this complex data web and translating it into a simple visual flow can be extremely challenging during large-scale modernization programs. The visual lineage graph adds tremendous value and helps define the roadmap to the modern data architecture. It deep dives into all the existing flows, like Autosys jobs, applications, ETL scripts, BTEQ/Shell (KSH) scripts, procedures, input and output tables, and provides integrated insights. These insights help data teams make strategic decisions with greater accuracy and completeness. Enterprises can proactively leverage integrated analysis to mitigate the risks associated with migration and avoid business disruption.
Now, let’s see how you can efficiently manage lineage.
To view the required lineage:
- Select either the Process or Data tab.
- Enter the keywords in the Search Keywords field.
- Click the Search icon to generate the lineage.
Process lineage illustrates the dependencies between two or more processes such as files, jobs, etc., whereas data lineage depicts data flow between two or more data-holding components such as entities, flat files, etc.
In addition, the filter search icon 
allows you to include or exclude particular nodes to obtain the required dependency structure. You can also choose the direction of the lineage. By default, the Dependency Direction is Left to Right Hierarchy. You can also choose Right to Left Hierarchy or Bidirectional dependency directions as required. Moreover, you can also increase the Hierarchy Levels to nth level.
Lineage facilitates you visualize how your selected nodes are connected and depend on each other. The nodes and their connecting edges (relationships) help you to understand the overall structure and dependencies.
|
Nodes |
Edges |
 Tables |
 Call |
 File |
 Read |
 Job |
 Execute |
 Autosys Box |
 Write |
 Script |
 OTHER |
Manage Lineage
This feature enables you to view and manage your lineage. You can add, modify, or delete nodes and their relationships to generate an accurate representation of the required dependency structure. There are two ways to update the lineage: either using Complete Lineage report or Lineage Template.
Using Complete Lineage report
Follow the below steps to modify the lineage:
- Click the Manage Graph icon.
- Click Download Complete Lineage to update, add, or delete the nodes and their relationships in the current lineage.
- Once the complete lineage report is downloaded, you can make necessary updates such as updating, deleting or adding the nodes and its relationships.
- After making the required changes, upload the updated lineage report in Upload to Modify Lineage.
- Click Apply to incorporate the updates into the dependency structure.
- Generate the required process or data lineage.
Using Lineage Template
Follow the below steps to add new nodes and their relationships to the current lineage report:
- Click the Manage Graph icon.
- Click Download Lineage Template.
- Once the lineage template is downloaded, you can add new nodes and relationships in the template.
- After making the required changes, upload the template in Upload to Modify Lineage.
- Click Apply to incorporate the updates into the complete dependency structure.
- Generate the required process or data lineage.
You can also apply:
| Feature | Icon | Use |
| Filter |  | Used to filter the lineage. |
| Reload |  | Assists in reloading graphs. |
| Save |  | Used to save the lineage. |
| Download |  | Used to download the file. |
| Expand |  | Used to enlarge the screen. |
Downloadable reports
Downloadable reports allow you to export detailed assessment reports of your source data which enables you to gain in-depth insights with ease. To access these assessment reports, click Reports.
Types of Reports
In the Reports section, you can see various types of reports such as Insights and Recommendations, Source Inventory Analysis, and Lineage reports. Each report type offers detailed information allowing you to explore your assessment results.
Insights and Recommendations
This report provides an in-depth insight into the source input files. It contains the final output including the details of queries, complexity, jobs, and so on.
Here, you can see the datastage folder, and DataStage Assessment.xlsx, and Lineage Dependency Report.xlsx reports.
DataStage Assessment.xlsx: This report provides insights about the source inventory. It helps you plan the next frontier of a modern data platform methodically. It includes a report summary, aggregated inventory, file summary, job summary, and more.
This report contains the following information:
- Report Summary: Provides information about all the generated artifacts.
- Volumetric Info: Presents a summary of the aggregated inventory after analyzing the source files. For instance, it provides volumetric info about the total number of files, jobs, shared containers, stages, queries, and likewise. It also provides job and query level complexity.
- File Summary: Lists all the input files along with the total number of jobs, components, complexity, and analyzed percentage. It also provides statistical information about the transformation components such as sequencer, Lookup, aggregator, and so on. These transformation components are transformed or converted to the target platform.
- Job Summary: Lists all the jobs associated with the input file. It also provides information about the complexity, analyzed percentage, the total number of components, and a lot more.
- Routines: Lists all the routines (Reusable custom script or pre-defined set of functions to perform specific operations or functionality in the DataStage job) present in the input files along with the file paths where the routine code resides.
- UnSupported Component: Provides information about the unsupported component types along with their frequency.
- Complexity Summary: Provides details about the complexity of jobs.
- Transformation Patterns: Provides details about transformations with similar patterns. It also includes information about transformation types, patterns, and occurrences.
- Job Patterns: Provides details about jobs with similar patterns. It also includes information about job types, patterns, and occurrences.
- Missing Job: Lists all the missing jobs along with their parent job.
Lineage Dependency Report.xlsx: This report contains information about views and job level lineages. It includes information about used and impacted tables, views, files, direct dependencies, dependency hierarchy and more.
This report contains the following information:
- view_report: Provides information about the views.
- script_report: Provides information about job level lineage.
Browse through the datastage folder to get datastage_column_lineage.csv, datastage_job_details_param.csv, job Analysis, Query.csv (CSV folder), job-level column lineage (datastage_column_lineage folder), and stage-level column lineage (datastage_column_stage_lineage folder) reports.
datastage_column_lineage.csv: This report provides information about column lineage. It includes information about source and target nodes (tables or files) along with associated columns, types, stages, and more.
datastage_job_details_param.csv: This report provides information about jobs including parent jobs, child jobs, parameters, values, and more.
Job Analysis.xlsx: This report provides information about jobs including the stages, stage types, container types, impacted and used tables along with a list of job parameters.
This report contains the following information:
- Job Analysis: Provides detailed information about jobs including the stage name, stage types, container types, impacted, used tables and more.
- Job Parameters: Lists all the job parameters.
Routines.csv: This report lists all the routines (Reusable custom script or pre-defined set of functions to perform specific operations or functionality in the DataStage job) present in the input files along with the file paths where the routine code resides.
Browse through the csv folder to view Query.csv report.
Query.csv: This report provides information about queries including the used and impacted tables, analyzed status, complexity, and more. If the analyzed status is Analyzed, it indicates that the query is analyzed successfully. Conversely, a Not Analyzed status indicates that the query is not analyzed.
Browse through the datastage_column_lineage folder to view job-level column lineage details. Each file includes information about a specific job such as source and target, along with their types, columns, stages, and more.
Browse through the datastage_column_stage_lineage folder to view stage-level column lineage details. Each file contains information about a specific stage (a component of a DataStage job), including associated jobs, source and target nodes along with their types, columns, and more.
Source Inventory Analysis
It is an intermediate report which helps to debug failures or calculate the final report. It includes all the generated CSV reports including external executable files, external procedures files, job details, unparsed files, invalid queries, and more.
assessment_unparsed Files.csv: This report lists all the unparsed DataStage files along with their type and the reason for parsing failure.
datastage_column_lineage.csv: This report provides information about column lineage. It includes information about source and target nodes (tables or files) along with associated columns, types, stages, and more.
datastage_job_details_param.csv: This report provides information about jobs including parent jobs, child jobs, parameters, values, and more.
invalid_query.csv: Lists all the invalid queries.
Routines: Lists all the routines (Reusable custom script or pre-defined set of functions to perform specific operations or functionality in the DataStage job) present in the input files along with the file paths where the routine code resides.
Browse through the csv folder to access External Executable File.csv, External Procedures File.csv, and Job Details.csv reports.
External Executable File.csv: Provides information about external executable files including jobs, executable file paths, commands, and more.
External Procedures File.csv: Provides information about external procedures files. It includes information about jobs, procedures, and its availability.
Job Details.csv: Provides information about jobs such as job types, child jobs, child job types, and more.
Lineage Report
This report provides complete dependency details for all nodes. It provides an end-to-end data and process lineage that helps to identify the complete dependency structure and the data flow.
This report contains the following information:
- Dependency (Process): Provides information about the process lineage.
- Dependency (Data): Provides information about the data lineage.
- Dependency (Data Model): Provides dependency details about the data models.
- Nodes: Lists all the source and target nodes along with its type.
- Volumetric Info (Summary): Provides volumetric information about the artifact types such as input tables, output tables, and DataStage jobs.
Leaplogic