What is Data Lineage?

Share on facebook
Share on linkedin
Share on twitter

In the world today, data has become the norm. It’s truly everywhere, but are we able to fully comprehend what our data is telling us? Are we even seeing the whole picture?


Probably not. Unless you’re using a tool that provides automated data lineage, the story you’re able to understand is, well, let’s just say incomplete. Why is that? Keep reading to find out.


In this article, you will learn:


Data Lineage Definition

We can define data lineage as the data’s life cycle or the full data journey. This life cycle includes: where the data originates, how it has gotten from point A to point B, and where it exists today. 


By utilizing data lineage, organizations can better understand what happens to data as it travels through different pipelines (ETL, files, reports, databases etc.). During its journey, data interacts with other pieces of information, is transformed, and is utilized in various reports. This allows businesses to make more informed decisions. It also enables companies to trace sources of specific business data in order to track errors and implement changes in processes as well as to streamline system migrations. This saves organizations significant amounts of time and resources, thereby tremendously improving BI efficiency and speeding up time to insights. Without understanding their data lineage tracking, companies aren’t able to predict the impact certain changes might have on various reports or ETL processes throughout the data environment. This means that they are dealing with an uncontrolled environment. This can be detrimental to a company because they can’t fully understand where their data came from and what happened along the way, nor can they extract value from their data.


How Is It Generated?

Now that we have established just how important it is to understand the origin and flow of a company’s data, you may be wondering how organizations can obtain this information? 


Well, that’s the problem – BI teams today tend to have to manually map out lineage since they are usually dealing with multi-vendor environments. For example, if they’ve got Informatica (ETL), Oracle (DWH) and Tableau (reporting), each has its own metadata labeling system managed by different teams. Trying to figure out where a specific data element in a Tableau report came from can be impossible, and if not completely impossible, you can bet it will take data analysts a really LONG time to figure out. 


Instead of manually compiling all of the data sources, companies can utilize an automated data lineage tool. This allows data teams to receive relevant information from all of their different data sources, instantly.


Impact Analysis in Octopai. Data lineage from ETL through DB and Analysis Services to Reporting.


What Does Data Lineage Look Like?

Data lineage is a visual representation of the overall flow of data. It provides a look at how data is manipulated via the ETL process. This allows organizations to assess the quality of their data before it is loaded into an analytics tool. Data lineage is primarily a visualization of the journey of different data points.


If you are still unable to picture this, refer to the image at the top of this post. It shows how Octopai’s automated BI intelligence platform compares and presents the lineage of two different reports (either from the same system or different systems – in this case it’s SSRS and Power BI). 


This clearly illustrates any variations between the reports and enables users to quickly understand exactly how two or more reports ended up showing differing results. Specifically, we see that an additional ETL process and table were found in the report on the bottom of the picture, which is missing from the report illustrated on the top. This is the point where the two reports began to diverge.


Data lineage also exists at two different levels – horizontal and vertical. The higher level view is horizontal because it shows the big picture of how data flows between systems. To drill in deeper, BI teams must look at the vertical view. They can sift through layers of data until they reach the column-to-column (or column-to-report) level. Vertical data lineage is helpful for such things as solving report discrepancies and getting a more comprehensive understanding of what exists in an environment that is going to be migrated to a new system. 

Sick of all the manual mapping required to sort out inconsistencies in your data?
Automated Data Lineage can fix that
Learn How


How Metadata Fits Into The Process

Whereas data lineage is the visual representation of how the data flows throughout various systems, the actual data presented in the lineage must first be located and verified. This is done through metadata management


Metadata is essentially the information about a company’s different assets and the relationships between them. With metadata, we are able to find all data items related to a specific report or ETL process, see all the dependencies related to it and then, trace its entire lifecycle.


In short, metadata is to data lineage what wheels are to a car. Metadata is what makes it possible to have data lineage, and the demand for tools to effectively manage metadata is growing rapidly.


Root Cause Analysis in Octopai. End-to-end data lineage from source systems to report through the entire BI landscape.

 

Introducing Data Lineage XD – Multilayered Data Lineage

Octopai’s Data Lineage XD goes beyond the standard one-dimensional data lineage. Instead, it offers three different layers of data lineage: cross-system lineage, end-to-end column lineage, and inner-system lineage. 


Cross-System Lineage:

Cross-system lineage provides end-to-end lineage at the system level from the entry point into the BI landscape, all the way to reporting and analytics. This type of lineage provides high-level visibility into the data flow – mapping where data is coming from and where it is going.


Top uses for cross-system lineage include:

  • Predicting the impact of a process change
  • Analyzing the impact of a broken process
  • Discovering parallel processes performing the same tasks
  • High-level visualization of data flow



End-to-End Column Lineage:

Cross-system lineage is perfect for the big picture. But what about when you need to zoom in and see the details? The end-to-end column lineage details column to column-level lineage between systems from the entry point into the BI landscape, all the way through to reporting and analytics. 


Top uses for end-to-end column lineage include:

  • Impact analysis of a change to a column in the source system
  • Root cause analysis to uncover the source of reporting errors
  • Column-level visualization of data flow
  • Regulatory compliance audit preparation



Inner-System Lineage:

Sometimes you need to dive even deeper into the nitty-gritty details of one particular system. The inner-system lineage details the column-level lineage within an ETL process, report, or database object. Understanding the logic and data flow for each column provides visibility at the column level no matter how complex the process, report, or object. 


Top uses for inner-system lineage include:

  • Visualizing the logic of a report, ETL, or database object data flow
  • Locating dependencies within a report


If you’re looking for the most in-depth, complete data lineage tool, look no further than Data Lineage XD.


Some Data Lineage Use Cases

Discovering Root-Cause of Reporting Errors. If the sales team is claiming a deal flow that simply doesn’t align with the Finance Department, you can be sure that the BI Manager will be asked to get involved. BI has to find out why the sales numbers are different than the finance numbers. They are able to visualize the entire data flow and determine root cause and impact analysis in just a few moments. With automated data lineage, BI teams no longer need to fear having to prove data accuracy in their reports. They can easily utilize data lineage to pinpoint the data in question and explain where it came from and any modifications it went through. Whether an error exists or not, BI professionals can feel confident in their explanation and provide this answer within a few minutes. Through the help of an automated data lineage tool, the business user will rest assured that all data is accurate and understood.


System Migration & Upgrades. Migrating seamlessly from a legacy BI tool to a modern one or upgrading to a new version of a system can be made significantly easier and streamlined by advanced data lineage that enables data teams to get full visibility into their BI environment. With automated lineage capabilities, teams can visualize which reports or ETL processes are duplicates, and which rely on data sources that are obsolete, questionable or non-existent so that they are able to reduce the number of data items that must be migrated – no need to migrate dups or obsolete reports, right?  Lineage visualization not only reduces time, effort and error in this process but also enables faster execution of the migration project. 


According to Forbes, lineage analysis helps identify “islands of data” that are not currently in use. This allows companies to understand the data they are actually utilizing and stop wasting money, time and effort on irrelevant stored information.


Data Privacy Regulations. When it comes to compliance, whether it’s GDPR, the California Privacy Rights Act (CPRA), or any of the numerous personal protection compliance acts on the horizon, you need to gain a better grasp of your data. In order to do that, you must have a data lineage tool in place. It is vital that you know where every single piece of your data originated. This is essential when it comes to protecting personal information. In order to remain compliant, data lineage can help the BI team identify a data element as PI so they can flag this and track all data items related to it. With this capability, companies will remain organized, transparent, and compliant. 


Impact Analysis. Before implementing a change, companies must understand what reports, data elements, or users will be affected. Through the use of automated data lineage, BI teams can identify the data objects downstream and see the potential impact. They can also pinpoint which business users interact with this data and how they will be impacted. By recognizing who and what will be influenced by this change, they can then decide if they should follow through with the modification.


You must also have a clear understanding of any change the data encountered along the way. Knowing the entire story behind each and every data item is a clear case of ‘knowledge is power.’  The more information an organization has regarding its data, the better prepared it will be for the future.  


Updated July 2021

Is your organization struggling to get full, accurate data lineage?
Octopai gets it done in 5 seconds, enabling BI groups to double their capacity
Schedule a Demo to See How

- See the power of data lineage xd - VIEW YOUR DATA FLOW FROM EVERY ANGLE​

We use cookies to ensure that we give you the best experience on our website. If you continue to use this site we will assume that you are happy with it.

Become a Partner

Well isn’t this exciting?! Thank you for thinking of Octopai! Please complete all form fields accurately so we can properly assess your request.

Partner Info
End Customer Info
End Customer Info (if known)

Thanks,
We got it!

One of our data lineage experts will be in touch soon.

In the meantime, be sure to check out our case studies showing how Octopai’s lineage transformed the way our customers work.