Thoughts & Writing

From the Blog

Technical deep-dives, opinions, and things I figured out the hard way — written the way I actually think about them.

Watermarks: The Unseen Guardians of Data Processing

I've always been fascinated by the behind-the-scenes mechanisms that keep our data pipelines running smoothly. One such mechanism that's particularly intriguing is the humble watermark. A watermark, in the context of data processing, is essentially a marker that indicates the pro

July 29, 20264 min read →

CDC (Change Data Capture)

Unpacking Change Data Capture: The Unsung Hero of Real-Time Data Pipelines

I've been diving deep into the world of data integration, and one technique that's really caught my attention is Change Data Capture (CDC). At its core, CDC is about identifying and capturing only the data that has changed in a source system. Sounds simple, but trust me, it's a g

July 29, 20264 min read →

Late-arriving Datadbtdata warehousedata enginner

Dealing with the Uninvited Guest: Late-Arriving Data

I've always found data warehousing to be a delicate dance between data ingestion, processing, and reporting. But what happens when the music stops, and the party's over, yet some guests still show up at the door? I'm talking about late-arriving data - the unwelcome visitor that c

July 29, 20264 min read →

data engineerdbtsnowflakeazure synaps

The Idempotency Mindset: Building Systems That Can Take a Punch

I've always been fascinated by the concept of idempotency. To me, it's more than just a technical term - it's a mindset, a way of thinking about system design that can make all the difference between a robust, fault-tolerant system and one that's prone to errors and data corrupti

July 29, 20264 min read →

MERGE vs UPSERTdata warehousedata engineer

The UPSERT Conundrum: When to Merge and When to Hold Back

I've always found myself pondering the nuances of updating databases, particularly when it comes to the age-old question: how do I efficiently insert or update records without duplicating effort? The answer, in many cases, lies in the realm of UPSERTs and MERGE statements. But wh

July 29, 20264 min read →

Incremental Loadsdata engineerdbtsnowflake

The Incremental Truth: Navigating Data Warehouse Loads

As I've worked with data warehouses, I've come to realize that handling full loads can be a daunting task. It's slow, expensive, and often feels like trying to drink from a firehose. One of the first solutions I learned as a junior was to filter data using SQL's `WHERE` clause, g

July 29, 20264 min read →

data engineeringdata warehouseSlowly Changing DimensionsETL

Embracing the Passage of Time: Slowly Changing Dimensions

I've always found it fascinating how data can change over time, and one concept that has particularly caught my attention is Slowly Changing Dimensions (SCDs). What's interesting here is how SCDs force us to consider the temporal nature of data, and the various ways we can choose

July 13, 20264 min read →

data engineeringdata modelingdata warehousestar schema

The Schema Showdown: Star vs Snowflake

I've spent countless hours designing data warehouses, and one of the most critical decisions I've had to make is choosing between a star schema and a snowflake schema. Both have their strengths and weaknesses, and I suspect that the right choice depends on the specific needs of y

July 12, 20264 min read →