🪄 Debezium: the magic behind data capture & async replication (for free)

July 22, 2024

This content originally appeared on DEV Community and was authored by adriens

Teaser

Did you ever find yourself in a situation where :

Team pushes data in a given database (let’s say MySQL,…) with its very own custom software
Team needs to get these data changes (INSERT, UPDATE, DELETE) as events so they can put them in let’s say… an another database (MariaDB, PostgreSQL,…)
Base software cannot be changed : you have to “deal with it”

Eg, team B’s motivation maybe to achieve datascience, RealTime Analytics, store in a datalake,…

This blog post is dedicated to this case… and surprisingly : open source solutions do exist to achieve this magic!

About the “why”

Debezium Project‘s “why” is pretty straightforward :

“Turn your databases into change event streams”

… even for “legacy” like systems:

// Detect dark theme var iframe = document.getElementById(‘tweet-1808144012127171020-834’); if (document.body.className.includes(‘dark-theme’)) { iframe.src = “https://platform.twitter.com/embed/Tweet.html?id=1808144012127171020&theme=dark” }

How it does NOT work (why it’s awesome)

The key thing here to remind is that Debezium does NOT act as a proxy in front of the database, and that’s the most elegant part.

The key point is that Debezium is literally listening to database changes, whatever you call them :

MySql binlog
MariaDB
PostgreSQL WALs for
Oracle archivelog
…

, then send these events in a common standard format into Kafka messages… waiting to be used later by one or many consumers.

How it works

The magic resides in the following workflow :

Capture data changes at the database level (WAL for postgres, archivelogs, whatever you call them…)
Send/Stream events to Kafka
Consume Kafka events so they they can be pushed to any third party data service 3′. JDBC : for example “consume events from multiple source topics, and then rite those events to a relational database by using a JDBC driver.”