Challenges of real-time streaming in Druid

Choosing Druid, which I wrote about in Why Druid, solved the query side of the problem. It did not solve the Kafka side. Getting a live stream of events into Druid correctly, without duplicating records, in a shape that could answer very different questions from the same underlying stream, turned out to be its own project, separate from anything inside Druid’s own configuration.

Four decisions did most of the work: where to enrich the data, where to deduplicate it, how to keep real-time segments from piling up, and where to aggregate it. Most of that ended up living outside Druid entirely.

At a glance

• Enrichment. Kept out of Druid and out of Kafka itself. A dedicated consumer read the raw topic, enriched each event, and published clean events to a second topic for Druid to ingest.
• Deduplication. Handled by that same consumer, checking each event’s key against Redis before forwarding it, so a duplicate never reached Druid at all.
• Segmentation and compaction. Hourly compaction merged the small segments real-time ingestion leaves behind into fewer, well-formed ones, cutting storage and simplifying every query that touched that data.
• Aggregation. The one job left inside Druid. Several datasources ran off the same clean topic, each rolled up along different dimensions, with the API layer routing each query to whichever one actually matched it.
• Outcome. Roughly 30,000 users at around 100 queries per second in steady state, later pushed past 200 in a separate tuning effort.

Enrichment: kept out of Druid, and out of Kafka itself

The raw events on the topic did not carry everything a query needed to answer a question. The obvious place to add that was inside the Druid ingestion spec, since Druid supports transforms at ingestion time. We decided against it, and against doing it as a heavy stream- processing job living inside Kafka itself. Either choice would have made Druid’s ingestion tasks or the Kafka cluster carry compute that had nothing to do with what each was actually built for.

Instead we built a dedicated consumer, a service external to both, that read the raw topic, enriched each event, and published the result to a second, clean topic for Druid to ingest from. That kept Kafka doing what Kafka does, moving messages, and kept Druid’s ingestion tasks doing what they do, building segments, with the enrichment work sitting in its own layer that could be scaled and changed independently of either.

Deduplication: the same consumer, backed by Redis

The same consumer handled deduplication, checking each event’s key against Redis before forwarding it: if the key had been seen recently, the event was dropped; if not, the key was written to Redis and the event went out enriched. Redis was the right tool specifically because that lookup had to happen on every single event at line rate, and it needed to stay fast enough that it did not become the new bottleneck we had just moved the work away from.

Druid’s own tools for this only apply after ingestion, through rollup and compaction collapsing matching rows later. Catching a duplicate before it is ever written is a cheaper fix than aggregating it away afterward, and it meant Druid never had to know a duplicate had existed at all.

Segmentation and compaction: cleaning up after real-time ingestion

Real-time ingestion from Kafka has a structural cost that has nothing to do with data correctness: Druid’s streaming ingestion hands off a new segment roughly every ingestion task and time chunk, which on its own produces a lot of small, suboptimally sized segments. Left alone, that adds up. More segments means more files for historical nodes to open and merge at query time, for no benefit to the answer.

We ran compaction on an hourly schedule specifically to clean this up, merging those small real-time segments into fewer, well-formed ones. That did two things at once. It compressed storage, since properly sized segments carry Druid’s columnar encoding and bitmap indexes far more efficiently than a pile of small ones. And it made querying simpler and faster, because a query touching an hour of data was now touching a handful of well-formed segments instead of whatever number of small ones ingestion happened to leave behind.

Aggregation: the one job we deliberately left inside Druid

Enrichment and deduplication were about cleaning the stream before Druid ever saw it. Aggregation was the opposite decision: that stayed inside Druid, because rolling up a stream along multiple dimensions in real time is exactly the job Druid is built for, and nothing upstream should try to replicate it.

The complication was that no single aggregation shape answered every question users asked. Rather than build one datasource and force every query to work around its shape, we ran several Druid ingestion specs off the same clean, enriched topic in parallel, each rolling the same events up along a different set of dimensions. The API layer decided, per incoming query, which of those datasources actually matched what was being asked, and routed to it instead of asking one general-purpose table to be the answer to every question.

What it held up to

In steady state, this pipeline served around 30,000 users with the query layer running at roughly 100 queries per second. Pushing that number past 200 queries per second was a separate, later effort, mostly on the ingestion and query-config side rather than the streaming pipeline itself, covered in Performance optimization for Apache Druid.

Lessons

The pattern across all four decisions was the same: put each piece of work in the layer actually built for it. Kafka moved messages. A dedicated consumer cleaned them. Redis remembered what had already been seen. Compaction kept the segments streaming ingestion left behind from piling up. Druid aggregated. None of those layers were asked to do another layer’s job, and that discipline mattered more than any individual optimization inside any one of them.