Migrating Data to the Cloud: 4 Critical Success Factors

Out with the old, in with the new. 

That means you should leave that old legacy system in the dust and migrate to a brand-spanking-new cloud-based data system as soon as possible, right?

Well, maybe.

There’s no denying the immense benefits that cloud computing can bring to data management: easier scalability, reduced cost of IT operations, improved performance… the list goes on.

But in order to reap those enviable benefits, you have to do something not-so-enviable:

Migrate. 

“Migrate” is a nice way of saying: Move. Displace. Uproot the data foundations of your organization and try to replant them. 

Data migrations aren’t simple, but if you make smart preparations and conduct the migration wisely, you’ll be able to celebrate with a bottle of champagne sooner than you think. 

4 critical elements to enterprise data migration success

1) Plan it out

If you migrate without a plan, and it actually goes well, you should thank your lucky stars. But since lucky stars are generally frowned upon as a risk management strategy, we highly recommend you plan out your cloud migration process.

Two types of planning are important:

• The high-level plan: migration goals
• The local-level plan: designing a migration to achieve those goals 

The high-level plan: migration goals

Why are you migrating? 

Don’t just point to the above list of migration benefits and say, “All of those!”

Take the time to consider your reasons, your goals, and your priorities among those goals. Get input from all the different departments of your organization that will be impacted by the data migration to the cloud.

If you skip this step (or do a half-hearted job), you’re leaving far too much to those lucky stars, just as you would in a physical move. Let’s say, for instance, that you suffer horribly from seasonal allergies and currently reside in Scranton, Pennsylvania (the “allergy capital” of the United States, according to the Asthma and Allergy Foundation of America). You want to relocate, and the number one thing on your mind is escaping that constant sneezing and those itchy eyes.

You see that Denver, Colorado ranks in the top 10 least challenging places to live with seasonal allergies. You do some research and are attracted by the scenic views, the recreational activities (no, not just the recreational substances) and the cultural opportunities. 

But you somehow overlook the fact (maybe because you’re at the peak of fall allergy season) that you hate shoveling out your driveway and sidewalks from Scranton’s annual 40 inches of snow. 

And guess what? After your move to Denver, you find yourself in a winter season where you need to shovel out 80 inches of snow! (Denver’s average is 60 inches annually, so maybe it’ll be better next year.) Maybe you should’ve considered Seattle. 

While such an extreme oversight is unlikely when it comes to a physical move (I mean, come on, who doesn’t picture Denver with snow-covered slopes?), it’s more of a risk when it comes to a digital or operational move. 

Listing and prioritizing your organization’s goals (and anti-goals, i.e. what you want to avoid) is the foundation of a successful cloud data migration.

The local-level plan: designing a migration to achieve those goals  

Once you have a high-level plan, it’s time to get down into the local picture. How are you going to pick a cloud infrastructure, design an ideal data environment and orchestrate a move to support those goals?

An important stage of this local-level planning is constructing a map of your current data environment and data flow – and then optimizing it. How would you ideally redesign your environment and data flow to achieve your goals? You can first do a no-holds-barred, data dream environment, and then (inevitably) tweak it to be more realistic and in line with your organization’s resources.

Creating a detailed map of your data landscape is prohibitive (and a huge time-suck) if you do it manually. That’s where automated data lineage can speed your cloud migration process along, by creating a comprehensive map of the data flow through your environment in as little as a day. (Okay, it *could* take a few days – if you have a massive environment.)  

Once you have a map, you can see exactly where your data flow and processes are tripping you up from achieving your goals, and you can design how an ideal flow and environment would look. 

Get feedback and buy-in from your stakeholders (so you can be sure you didn’t overlook 60 annual inches of snow). Then use that approved design as your guiding light in structuring your migration.

2) Don’t move junk

This may sound obvious. After all, why would you pay to move junk and then pay to store it?

But when you’re looking at an overwhelming project like a cloud migration, it is tempting to simplify the process wherever you can. 

So whatever data you can migrate as-is (especially if you’re doing a lift and shift migration, or even a “lift, tinker and shift” migration), you may want to just do it and relegate sorting through it until a later point in time, after the migration.  

Resist the temptation. 

Sorting through your data assets and processes has a direct impact on the ROI and success of your migration. 

I know someone who was born and grew up in the United States but never lived there permanently as an adult. She went to college abroad… and decided she wanted to stay abroad to live. Her “move” (if you can even call it that) was seamless, and consisted of placing in her suitcase whatever clothing or childhood items she wanted from her parents’ home during her vacation visits back. 

Her parents, on the other hand, joined her years later as an almost-retired couple who had lived their entire adult lives in the United States – and their move was a move. They had, naturally, accumulated the kind of “stuff” you amass when you live in the same general geographic area for decades.  

But now they were moving overseas, requiring a shipping container where you pay per cubic foot of space. They were moving from a three-story private home into a three-bedroom apartment; if they moved all their possessions “as-is,” they’d have to rent storage space – money out of their pockets PLUS limited accessibility to their possessions.

So it was time to downsize.  

It took time. It took work. But making the move to a new country lighter, freer, with only the possessions they really wanted and needed, had positive impacts financially and logistically. Getting settled and feeling at home in your new environment takes less time if you have less stuff.

So even if you’re doing a lift and shift migration – and all the more so if you’re revising, rebuilding or replacing your current data management system – invest the time and effort to go through your current assets and processes and see what should really be making the move. 

And it may not be as much time and work as you expect. If you can leverage automated data lineage to trace the path of every data asset end-to-end through your systems, from source, to ETL, in the database, through to analytics and reporting, you’ll have an invaluable, visual map in a short time. You’ll be able to see clearly which assets and processes are valuable, which are discardable, and which need to be tweaked or fixed. 

With the full picture at your fingertips, you can wisely pack for a successful move.

3) Learn the language and culture

Similarities can be deceiving. 

A British citizen who moves to the United States thinks that the language will be the least of his difficulties. That is, until he is half under his bed, searching for a missing sock, and asks his roommate to hand him a torch. Instead of said roommate’s flashlight, he gets a lecture about fire safety. And then, when he asks around for a place to get braces, he’s directed to an orthodontist instead of the nearest clothing store that carries suspenders. 

Oops. 

Your legacy data systems, no matter how similar they seem to the cloud data systems to which you are migrating, will have some cultural differences. Be prepared.

These differences might be as deceptively simple as a syntax change. For example, Instr () in Oracle db needs to be written as Position () in Snowflake. That one’s actually not so bad, because Snowflake just won’t support the Instr () query structure, and you’ll realize there’s something wrong.

But what about this query: select concat(‘SSS’,null) from dual? Because of the difference with how they process the CONCAT function, Oracle db will return SSS, while Snowflake will return null. 

Ask for a torch, and instead of getting a flashlight, you get handed the burning brand they use to light the main fire at the Olympics. And you may not even realize until you use it to search under your bed for your missing sock. 

Your bad.

To avoid wildfires in your data environment, do thorough research on the differences between your legacy systems and your new cloud systems. Train anyone who will have interaction with the new system on the nature of these differences – before your ETL, reporting system or database migration to the cloud. 

Data analysts who will be running queries across Snowflake’s pay-for-compute-time platform, for example, need to understand the cost and impact of different types of queries. They need to be briefed on the kind of queries NOT to run – especially if those were standard queries on your legacy system.  

An ounce of prevention is worth a pound (is that lb.? or £?) of cure.

4) Minimize day-to-day disruption

The last time I moved apartments, I found myself still (barely) awake at midnight, dragging myself around looking for a box containing some critical items. (My spouse, of course, did not consider the items as critical as I did and saw no need to extricate himself from the bed to help me search.) Needless to say, I was NOT in a good mood.

Cranky employees tend to have a negative impact on your ROI. If you can maintain your employees’ ability to easily find and access the data they need, even in the midst of your migration, the migration will go much more smoothly. (Or at least it will feel like it’s going more smoothly, which often amounts to the same thing.) 

An automated data catalog can be an invaluable aid to keeping data assets and their location at your employees’ fingertips. The data catalog itself is a tool that organizes all the data assets in a company’s data landscape, for each one including definitions, descriptions, ratings, responsible individuals, and more, making it simple to search for, identify and access the data you need for any given purpose.

When a data catalog solution is automated, it will automatically extract the data assets from across the different software in your data landscape, including ETLs, databases, and reporting tools. An automated solution can collect, analyze, and organize – and then build or populate the data catalog by itself. 

An automated data catalog will not only self-create but will also self-update, routinely reviewing all metadata within your data landscape and updating your data catalog accordingly. At no time is this more important than during a migration. Your automated data catalog can review both your on-prem data environment and your new cloud environment and integrate both into your catalog. Frequent automated updates will make the current location of any data asset just a quick search and click away. (Similarly, if your end goal is a hybrid environment, a comprehensive automated data catalog can help keep everything straightforward and accessible, no matter where it’s located.)

Result: no service disruptions. Your data-dependent employees may not even realize you’re in the midst of a migration. 

Shout it (c)loud!

You’ve done it! You’ve made it successfully to the cloud! 

Well, probably you’ve only made it to the end of this blog post, but envisioning success is supposed to be helpful for athletes, so why not for data experts?

Try this: envision yourself taking that bottle of champagne and celebrating the launch of your new cloud-based data management system!

Did it work? Now go get ‘em, champ – get that data to the cloud!