Azure SQL & SQL Server Change Stream with Debezium

Responding to data changes in real time using modern technologies

Cosmos DB Change Feed or MongoDB Change Stream are an easy-to-consume version of Change Data Capture

In fact, both the named databases offer a cool feature called Change Stream in the first and Change Feed in the second, that are basically a simplified concept of the aforementioned Change Data Capture, with a big, important, difference: the stream is easily accessible via specialized API that follows a pub/sub pattern, allowing any application (and more than one at the same time) to be notified when something has changed so that it can react as quickly as possible to that change.

The ability to get the changes that happens in a database and send it to a message bus for near real-time consumption is becoming a base requirement.

As pattern like CQRS and Event Sourcing are becoming more and more popular (see the Reactive Manifesto), and microservices architecture are all the rage, the ability to get the changes that happens in a database, nicely wrapped into a JSON or AVRO message, to be sent into a message bus for near real-time consumption is becoming a base requirement.

Apache Kafka is used for building real-time data pipelines and streaming apps.

It’s now time to have SQL Server, Change Data Capture and Apache Kafka working together to provide a modern, easy to use, Change Data Capture experience.

The magical black-box

On Azure, Event Hubs can be used instead of Apache Kafka, to make the solution even simpler

You’re really not an Apache Kafka person, but still like the idea of having a stream of changes and a microservices architecture? I hear you: that’s why, in the sample available on GitHub, instead of Apache Kafka I’m using Event Hubs: all the nice things without the burden of maintaining an entire Apache Kafka cluster.

You have the best of both worlds: a proven, flexible, transactional database and modern, reactive, event-driven developer experience

If, instead, you want to understand how everything works behind the scenes, read on.

Inside the black box

Debezium is actually a Kafka Connect extension: it uses Kafka Connect API to store and manage configuration and metadata.

Where can I use this?

Honestly? Everywhere! The stream-processing (or event-based, reactive, architecture) approach is applicable everywhere and it will become more and more popular as soon as more people start to give near real-time updates and actionable insights for granted. No more batch updates.

--

--

Data Geek, Storyteller, Developer at heart, now infiltrated in Azure SQL product group to make sure developers voice is heard loud and clear. Heavy Metal fan.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Davide Mauri

Data Geek, Storyteller, Developer at heart, now infiltrated in Azure SQL product group to make sure developers voice is heard loud and clear. Heavy Metal fan.