Data streaming vs batch processing

You stood up Kafka to generate a daily report

I've seen this movie more than once: the team needed a report that ran once a day, overnight, and someone decided the "modern" solution was a streaming pipeline. Kafka, consumers, offset management, deduplication, late-event handling — the full artillery. Six months later, three engineers spent half their time babysitting streaming infrastructure to deliver a number that a cron with one query would have produced in fifteen minutes. The required latency was hours. They built for milliseconds.

This piece compares batch processing and data streaming, and helps you choose without getting swept up in the hype. First, a practical prerequisite for both worlds: processing volume without blowing up RAM — on that it's worth reading how to handle large files without blowing up memory.

What batch processing is

Batch operates on bounded data: a set with a known beginning and end. You gather a period's data, run the processing all at once, and produce the result. It's the model of the nightly report, the scheduled ETL, the monthly close.

The properties that matter: high throughput, because you optimize for processing large volumes at once; simplicity, because the data is at rest and you know exactly how much there is; and high latency, measured in minutes or hours, because the result only comes out when the whole batch finishes. You trade response time for simplicity and efficiency. For a report nobody looks at before nine in the morning, that trade is excellent.

00:00  cron fires
00:01  reads all of yesterday's orders (closed set)
00:14  aggregates, computes, writes the result
00:15  done — latency: ~9 hours until anyone reads it

What data streaming is

Streaming operates on unbounded data: a continuous flow that never "ends." Events arrive, are processed one by one (or in small windows), and the result updates in near real time. It's the model of fraud detection, the live dashboard, the alert that needs to fire now.

The advantage is low latency: the result tracks the event. The cost is that streaming is genuinely harder, and the difficulty isn't accidental, it's in the nature of the problem:

• Ordering. Events can arrive out of order. The click that happened first can be processed later.
• Late events. An event from ten minutes ago arrives now, after you've already closed the window for that period. What do you do with it?
• Exactly-once. Guaranteeing each event is counted exactly once, even with failures and reprocessing, is one of the thorniest problems in distributed systems.

None of these problems exist in batch, because in batch the set is closed and at rest.

When each one fits

The right question isn't "which is better," it's "what latency does the business actually require." The answer dictates the choice:

• Batch for nightly reports, ETL, accounting close, ML model training, monthly billing, anything where "tomorrow morning" or "in an hour" is an acceptable answer.
• Streaming for fraud detection, live dashboards, operational alerts, real-time recommendation, monitoring — cases where a result ten minutes late is already worthless.

The mental test I use: if the result is ready an hour later, does someone lose money or make a wrong decision? If not, it's batch. Fraud detection an hour late is useless; a sales report an hour late is exactly what's expected.

The complexity tax of streaming

Here comes the opinion that tends to bother people: most teams reach for streaming long before they need it. Streaming is seductive — it sounds modern, scalable, real-time — and so it enters the architecture decision by status, not by requirement.

The problem is that streaming charges a permanent complexity tax. You start dealing with state management, checkpoints, watermarks, windows, delivery guarantees, reprocessing — a whole class of bugs that simply doesn't exist in batch. Each of those pieces is a place the system can fail at 3 a.m. A batch pipeline that fails you re-run; a streaming pipeline that fails demands understanding which offset it stopped at, whether there was duplication, and how to recover state.

My rule: start with batch. Migrate to streaming when batch latency provably costs money or breaks a requirement, not when someone thinks it would be cool. Streaming is a powerful tool for the right problem and dead weight for the wrong one.

Micro-batching: the middle ground

Between the two extremes sits micro-batching: processing small batches at short intervals, every few seconds, instead of event by event. Spark Structured Streaming popularized the approach. You get latency in seconds — good for most "near real-time" cases — without paying the full complexity cost of pure event-by-event streaming.

For many dashboards and operational metrics, five-second latency is indistinguishable from real time to the user, and micro-batching delivers it with a mental model much closer to batch. Before adopting pure streaming with all its expensive guarantees, ask whether a batch every few seconds solves it. In practice, it solves more cases than people assume.

Frequently asked questions

Does streaming always have lower latency than batch?

Generally yes, that's streaming's reason to exist. But the lower latency comes with a complexity cost that isn't always justified. If the business tolerates minutes or hours, streaming's lower latency buys nothing of value and you pay the tax for nothing.

Can I start with batch and migrate to streaming later?

You can, and it's usually the more sensible path. Start with batch, prove that latency is a real bottleneck, and only then migrate. Rewriting a working batch pipeline as streaming is cheaper and safer than debugging a premature streaming system you never needed.

What makes exactly-once so hard in streaming?

Because in a distributed system with failures, guaranteeing each event is processed exactly once requires coordinating state, offsets, and idempotency across restarts and reprocessing. In batch the set is closed, so counting once is trivial. In a continuous flow, partial failures and redeliveries make this a fundamental problem.

Is micro-batching batch or streaming?

It's a middle ground. Technically it processes small batches at short intervals, with a mental model close to batch, but delivers latency in seconds like streaming. For most "near real-time" cases, it's the balance point between simplicity and low latency.

How to decide in practice

Batch and streaming don't compete for the same problem — they solve problems with different latency requirements. Batch delivers simplicity and throughput for bounded data when minutes or hours suffice; streaming delivers low latency for continuous flows when delay costs money, at the price of real complexity. Don't choose by hype: define the latency the business requires and let it decide. When in doubt, start with batch, consider micro-batching as a middle ground, and only move to pure streaming when the requirement proves you need it.