top of page

What is Change Data Capture Anyway?!?



Alright, data engineers, gather around, and let's crack this nut with a bit of Adam Sandler humor: Why did the database developer break up with the data warehouse? Because they couldn't handle the load and were always out of sync! 😆 But seriously, folks, if you're in the trenches of data engineering, you know our world could use some good old CDC - and no, I'm not talking about the health agency.


What is Change Data Capture Anyway?!?

So, here's the deal. Picture this: Your production databases are like those high-maintenance friends who have all the coolest stuff you want to borrow (aka analyze), but you don't want to be the reason their life (or performance) falls apart. That's where Change Data Capture (CDC) slides in, cooler than a fresh pair of kicks, saving the day without adding drama to your production databases.


What is the problem we are solving?

It turns out, at scale, this is a hard problem to solve. You can’t just decide to copy data over from the production database to the warehouse—that would add a lot more load on the production database, especially if you want high fidelity. And if you fetched only the changed records, you would miss deletes.

Thankfully, all modern production databases write out a write-ahead log (WAL), or change log, as part of their normal transaction processing. This log captures every single change to each row/cell in each table in the database and can be used in database replication to create replicas of the production database. In the CDC, a tool reads this write-ahead log and applies the changes to the data warehouse. This technique is a lot more robust than batch exports of the tables and has a low footprint on the production database.


Let's use an easy-to-imagen example (CDC's it)

Imagine trying to move a sofa (your data) from one apartment (your production database) to another (your data warehouse) without blocking the hallway (slowing down your database). Directly lifting that sofa can cause a scene, especially if it's as busy as Times Square on New Year's Eve. But what if you could magically teleport each piece of the sofa without anyone noticing? That's CDC for you, using the magic of the write-ahead log (WAL), a secret diary where databases jot down every little change they make, from a new cushion (data insert) to throwing out old coffee stains (deletes).

But here's the kicker: running a CDC operation is like planning a heist. You need the right crew (tools) and a solid plan to handle, here is a partial list:

Scale

Imagine if your diary got so full, that your bookshelf collapsed. That's what happens if the CDC can't keep up with the WAL, and suddenly, you're out of disk space.


Replication lag

Ever tell a friend you'd text them when you leave but forget until you're already there? That's replication lag. You need those texts (data) to be timely, or your friend (data warehouse) gets left behind.


Schema changes

It's like if your friend suddenly decides they're into minimalist furniture after you've already bought them a baroque sofa. You've got to keep up with their tastes (database schemas) or you're giving gifts that don't fit.


Masking

Sometimes, you've got secrets (sensitive data) you don't want the new apartment's landlord (compliance regulations) to know about. Masking helps keep those secrets while still moving the sofa.


Historical syncs

Before you start teleporting pieces of the sofa, you need to make sure the entire thing is ready to go. That's your historical sync, making sure nothing's left behind before you start with the day-to-day.


Now, you might think, "Hey, I could build my magic teleportation device (CDC connector)!" But let's be real, would you rather build a teleporter from scratch or grab one off the shelf that's got a warranty? Use the tools out there, folks. They're tried, and tested, and won't leave you stranded in another dimension.

So, next time you're looking at your precious production databases, think of CDC as your secret agent, working behind the scenes to keep everything cool, collected, and most importantly, in sync.


Real-life example, here you go

LinkedIn uses CDC to manage the enormous volume of data generated by its millions of users. They've got this down to an art form, ensuring that every job change, new connection, or post update is captured in real time and reflected accurately across their analytical data stores. This isn't just moving data; it's about keeping the pulse of the professional world-beating in harmony.

By employing CDC, LinkedIn ensures that their data lake is always stocked with the freshest data, without overburdening their production systems. It's as if they're conducting a symphony of data, with each instrument (database) perfectly in tune, allowing them to deliver insights, recommendations, and connections that feel as personalized as a hand-written letter in a digital age.

So, the next time you update your profile or make a new connection, remember, that there's a whole world of CDC magic working behind the scenes, making LinkedIn a bustling, ever-up-to-date hub of professional activity.


And there you have it, fellow data adventurers: with the right tools and a bit of stealth, navigating the world of CDC is like pulling off the perfect data heist. So gear up, embrace the night, and let's make our data move as smoothly as a cat burglar in a world where the only trace left behind is the success of our seamless integrations. Here's to mastering the art of data replication, one covert operation at a time!


Comments


bottom of page