Quick and Easy Migration to Azure Data Factory from a Legacy ETL

The Data Challenge

Data & Analytics teams must understand which processes to migrate to Azure Data Factory as not all are still relevant. Focusing on the processes that are actually in use can significantly reduce the migration load, and later, the maintenance and cost given that ADF charges by usage. The BI team must answer the following questions to ensure a pain-free, cost-effective migration:

Which processes are no longer in use?
Are there processes that were needed for a one-time or short-term purpose and are no longer relevant?
Has the data underlying the process changed in some way, and can no longer be relied upon?
Are there processes based on legacy data that is no longer being updated, and therefore not useful anymore
Which processes completely duplicate one or more other processes?
Which dependencies need to be created between the new processes? Accurate data many times relies on the upstream processes running in the correct sequence. The timing of each step needs to be analyzed and set accordingly.
And finally, once the migration is complete, how can we make sure that all of the necessary processes successfully migrated?

How Data & Analytics Teams Worked Before Octopai

Data & Analytics teams are accustomed to taking a manual approach when it comes to migrating data systems and answering the above questions. This would normally mean going through the portfolio, process by process, in order to:

Find and eliminate processes that simply aren’t needed anymore
Identify redundancies, obsolete, or unreliable data sources and targets
Target ETL processes that fail to capture relevant data or combine it ineffectively with other data

It can require endless amounts of detective work, combing through logs and job schedules in disparate systems and examining stored procedure code, report definitions, and more.. The process usually takes many months. And because the company’s data landscape is naturally always changing, Data & Analytics teams ends up chasing a moving target.

How Octopai Empowers Data & Analytics Teams

Octopai’s automated data lineage and discovery platform takes the painful manual work and inevitable errors out of the preparation process for migrating to Azure Data Factory. How?

The visual data lineage tools can show which ETL processes are relevant. Specifically, which processes rely on data sources that are obsolete, questionable, or nonexistent as well as targets that are simply no longer in use; and more.
ETL processes can be identified and analyzed to ensure they are generating the expected output in the right sequence.
The BI team can easily see the relationships between different data sources, greatly simplifying the migration process.

Octopai is the first Data Intelligence Platform in the industry with the unique capability of visualizing the Azure Data Factory pipelines’ full column-level, source-to-target traceability. This is shown through different data transformations at the most detailed level. As with all other tools that Octopai supports, enterprises can now view complete end-to-end data lineage, from original source systems through Azure Data Factory all the way to reporting.

Value to the Organization

Significantly reduced time, effort, and error in preparing for migrating to Azure Data Factory
Assurance that only the ETL processes needed are migrated leading to significant cost savings to the enterprise since, as mentioned above, ADF charges per usage.
Faster execution of the migration project, enabling businesses to take advantage of Azure Data Factory much sooner.

And what happens after the migration to Azure Data Factory is complete? Your Data & Analytics teams will be able to:

Easily access column-level data lineage within ADF to ensure that everything migrated successfully.
Enable employees that are not yet familiar with ADF and its processes to visualize the lineage without in-depth ADF training.
Visualize complete end-to-end data lineage from ADF through to reporting, automatically, in seconds for ongoing day-to-day BI challenges such as impact analysis, root cause analysis, and more.