Why Druid: building a real-time analytics product for tens of thousands of users

In late 2020 I started planning a new architecture for a problem that looked simple on the surface and turned out to be anything but: give thousands of users a live, 360-degree view of operational performance, and make it fast enough that nobody ever waits for a number.

The existing approach leaned on a traditional enterprise BI and reporting stack, the kind that a lot of large organizations still run on. It worked, but it was hitting a ceiling. Leadership pushed a good challenge at me: stop treating this as another report, and build it as a real product. That reframing changed everything. I was no longer wiring up dashboards. I was designing a data-heavy software product that had to behave like an application, not a nightly batch job.

The requirements that shaped everything

Once I wrote the constraints down, the hard part became obvious. The product had to:

• Serve tens of thousands of users with interactive, concurrent access, not a handful of analysts.
• Return summaries from very large volumes of data in milliseconds, not seconds. Aggregations over huge fact tables, sliced across many dimensions and time, needed to feel instant.
• Ingest transaction data in near real time, so the numbers reflected what was happening now, not yesterday.
• Stay cost-efficient and horizontally scalable, because both the data volume and the user base were only going to grow.

That combination is the whole story. Plenty of databases can do one or two of these well. Very few do all of them at once. High concurrency, millisecond aggregations, and streaming ingestion pull a system design in different directions, and the traditional reporting stack simply was not built for it.

The evaluation

I did not want to guess. I prototyped against the workload and put four options through their paces.

MongoDB. A great document store, and easy to move fast with. But this was never a document problem. It was an analytical aggregation problem over wide, high-cardinality data. Large group-by and roll-up queries at our concurrency were not where a document database wants to live, and I would have spent my life hand-tuning indexes and pre-computed collections to fake an OLAP engine.

Cassandra. Excellent for write-heavy, key-based access at scale, and the ingestion side was tempting. The problem is that Cassandra rewards you for knowing your queries in advance and modeling a table per access pattern. Ad-hoc, multi-dimensional slice-and-dice over billions of rows is the opposite of what it is designed for. Flexible analytical aggregation was a fight, not a feature.

HBase (with Phoenix). A serious contender. Phoenix gave us SQL over HBase, and the scale story was real. But to hit millisecond summaries we would have had to build and maintain our own aggregation and pre-rollup layer on top, plus wear the operational weight of the HBase and HDFS stack. It was possible, but it was a lot of custom engineering to reach a place another tool started from.

Apache Druid. This is where the workload and the database finally agreed.

Why Druid fit

Here is the detail I should have led with: this is fundamentally a time-based problem. The domain is a stream of events over time, and almost every question is a time-sliced aggregation, such as activity in the last hour, today versus last week, or a trend across a season. Druid is a real-time, time-series-oriented analytical (OLAP) database, so the data model and the questions were a natural match rather than something I had to force. Almost every one of its design choices lined up with my requirements:

• Columnar storage with time-based segments. The data is partitioned by time and stored column by column, so a query that aggregates a few columns over a date range only touches what it needs. That is exactly the shape of the questions we were answering.
• Real-time Kafka ingestion while serving queries in milliseconds. Druid ingests directly from Kafka and makes events queryable within seconds, and it does this at the same time as it answers thousands of user queries in milliseconds. Ingesting and serving are not competing modes you switch between; they run together on the same live data. That single property was the clearest signal that Druid was built for this use case.
• Batch ingestion alongside the stream. The same system also handles historical batch loads, so backfills and corrections sit next to the live stream without a second architecture.
• Roll-ups and pre-aggregation at ingestion time. By aggregating as data lands, the volume the query engine has to scan drops dramatically. This is the single biggest reason millisecond summaries over large data became realistic.
• Bitmap indexes and fast filtering. High-cardinality filters across many dimensions are handled with inverted bitmap indexes, so slicing stays quick even as dimensions explode.
• A scatter-gather, distributed query model built for concurrency. Druid fans a query out across data nodes and merges the results, which is what let the design credibly aim at tens of thousands of concurrent users rather than dozens.

None of the other candidates were bad databases. They were simply built for different problems. Druid was the one that treated “fast aggregations over a lot of streaming data, for a lot of people” as its core job instead of an add-on.

What I took away from it

Two things stuck with me. First, the most valuable move was writing the real constraints down honestly before touching any technology. The requirements chose the database; I just confirmed it. Second, reframing the work from “a report” to “a product” was what unlocked the right level of engineering. When you commit to application-grade latency and scale, you stop patching a reporting tool and start designing a system.

Druid proved to be the right foundation for this use case. If you are facing a similar mix of real-time ingestion, heavy aggregation, and high concurrency, it is worth a serious look before you try to bolt an OLAP layer onto a database that was never meant to carry one.