Reporting Kafka and Spark streaming metrics with a custom Spark listener

I run Kafka to Spark streaming pipelines, and I had a blind spot that bothered me: I could not easily see how the streaming jobs were doing. The jobs run continuously, but there is no shared dashboard for their health. The Spark UI shows some of it, per application and only while you are looking, and Kafka to Spark streaming metrics are not published anywhere the team can watch over time. This is not solved out of the box, so I built it.

The gap

Streaming jobs are long-running, so the questions you ask about them are operational, not one-time:

• Are we keeping up? Is processing time getting close to the batch interval, which is the warning sign that the job is falling behind?
• Are batches completing on time, or is scheduling delay creeping up?
• What is the throughput, and how much is each batch pulling from Kafka?
• Did something fail, and when?

The Spark UI could hint at some of this, but it is per-application and ephemeral. It is not a dashboard you can leave on a wall, trend over days, or hand to the rest of the team. I wanted continuous, shared visibility.

The approach: a streaming listener

Spark exposes a streaming listener interface, and that turned out to be the clean hook I needed. It fires callbacks as the streaming job runs, and it hands you the metadata Spark has already computed for each batch. I wrote a listener that captured that metadata and published it out, without touching the job logic itself.

The metrics worth capturing were the ones that describe streaming health:

• Batch duration and processing time, so I could compare them against the batch interval.
• Scheduling delay and total delay, which reveal a job starting to fall behind before it actually breaks.
• Records per batch and processing rate, for throughput.
• Input offsets per batch, so I could see how much each batch pulled from Kafka.

The listener pushed all of that into InfluxDB. Because it is a time-series database, streaming metrics that are naturally stamped in time fit it cleanly, and every batch became part of a growing history rather than a line in a log that scrolled away.

Grafana for the view

On top of InfluxDB I built Grafana dashboards so the whole team could see the streaming pipelines live and historically: throughput over time, processing time against the batch interval, and scheduling delay trends. Instead of one person tailing logs, anyone could open a dashboard and tell at a glance whether the pipeline was healthy or starting to slip.

Why it is custom

The reason this takes real work is that, as far as I can tell, it is not available out of the box. Spark has a metrics system with sinks, but a ready-made Kafka to Spark streaming view, with batch health and throughput published to a time-series store and charted for a team, is something you wire up yourself. The listener interface is the right seam to do it: hook the numbers the engine already produces, send them somewhere durable, and visualize them.

What it buys us

The payoff is early warning. We can see backpressure forming as processing time creeps toward the batch interval, before it turns into a real incident. Just as important, streaming health stops being tribal knowledge and becomes something the whole team can see.

The pattern is simple and it keeps earning its keep: hook the metrics the system already produces, store them as time series, and put a dashboard in front of people. Spark may add richer streaming metrics of its own over time, but until then, the listener is a clean way to get real visibility today without waiting for it.