8. Streaming: data that never ends

Streaming processes unbounded data continuously, and it rests on one abstraction: the append-only log. Its hard problems are all about time: events arriving late, out of order, and needing memory (state) to make sense of them.

Stream only when freshness pays

Streaming is harder to build, test and operate than batch. Use it when the value of data decays fast: fraud detection, live recommendations, monitoring, or keeping a RAG index current. Otherwise, a batch job every hour is usually the better engineering choice.

Freshness is a spectrum: daily batch, hourly batch, micro-batch every few seconds or minutes, and true streaming in milliseconds. Each step costs more in complexity and money. Ask what decision actually changes with fresher data.

The log

A log is an ordered, append-only sequence of records, each with a position number called an offset. Producers append to the end. Each consumer reads at its own pace and remembers its own offset. Records stay for a retention period, so a consumer can rewind and replay history.

You already met this idea as the database write-ahead log in section 4. A log is the universal way to move changes between systems: one writer, many independent readers, and a replayable history.

Apache Kafka concepts

• Topic: a named log, such as “page_views”.
• Partition: a topic is split into partitions for parallelism. Order is guaranteed only within a partition.
• Key: records with the same key go to the same partition, so all events for one user stay in order.
• Consumer group: the partitions are divided among the consumers in a group. Parallelism is capped by the number of partitions.
• Retention and compaction: keep records for N days, or compact a topic to keep only the latest record per key, turning it into a table.
• Replication factor: how many brokers hold a copy of each partition.

Alternatives with the same model include Redpanda, Apache Pulsar, Amazon Kinesis and Google Pub/Sub. A classic message queue (RabbitMQ, SQS) differs: it deletes a message once consumed, so there is no replay.

Event time versus processing time

• Event time: when the thing actually happened, stamped at the source.
• Processing time: when your system sees it.

They differ constantly. A phone goes offline in a tunnel and uploads two hours of events at once. A network hiccup delays one partition. If you aggregate by processing time, your “clicks per minute” chart is wrong whenever anything is delayed. Correct systems compute on event time.

Windows

Aggregating an endless stream requires cutting it into finite windows.

Window	Shape	Example
Tumbling	Fixed size, no overlap	Orders per 5-minute block
Sliding (hopping)	Fixed size, overlapping	Transactions in the last 10 minutes, updated every minute
Session	Ends after a gap of inactivity	A user’s visit, ending after 30 idle minutes

Watermarks and late data

In event time, when is the 14:00–14:05 window complete? You can never be certain another late event will not arrive. A watermark is the system’s running estimate: “events older than T have probably all arrived”. When the watermark passes the end of a window, the window’s result is emitted.

This is a direct trade-off between latency and completeness. An aggressive watermark gives fast but sometimes incomplete answers; a cautious one gives complete but slow answers. Events arriving after the watermark are late data: drop them, send them to a side output, or update the earlier result.

State

Counting, aggregating and joining streams all need memory of what came before. Stream processors keep this state locally, often in RocksDB, and periodically checkpoint it to durable storage. After a crash, they restore the last checkpoint and rewind the input log to the matching offsets.

Checkpoints plus replayable input plus transactional output is how engines such as Apache Flink achieve effectively-once results. Watch state size: a join that remembers every event forever will eventually exhaust memory, so set expiry times.

Engines

Engine	Model	Known for
Apache Flink	Record-at-a-time streaming	Strong event-time handling, large state, low latency
Spark Structured Streaming	Micro-batches	Same API as Spark batch
Kafka Streams	A library inside your application	No separate cluster to run
Streaming databases (Materialize, RisingWave)	Incrementally maintained SQL views	Streaming written as plain SQL

Stream-table duality: a table is a snapshot of a stream at a moment, and a stream is the changelog of a table. Holding both views in your head makes most streaming designs easier to reason about.

Change data capture (CDC)

CDC reads a database’s own write-ahead log and publishes every insert, update and delete as an event. Debezium is the best-known open-source tool.

It beats the naive alternative, polling “WHERE updated_at > last_run”, in three ways. It captures deletes, which polling cannot see. It never misses intermediate updates. And it puts almost no load on the source database.

CDC is how you keep a warehouse, a search index or a vector index in sync with an operational database. When a support article is edited, CDC triggers re-chunking and re-embedding of just that article.

Lambda and Kappa architectures

• Lambda: a batch layer for correctness plus a streaming layer for speed, merged at query time. Powerful, but the logic is written twice.
• Kappa: everything is a stream. To reprocess, replay the log through new code.

Today’s common pattern blends them: stream data into lakehouse tables continuously, and run batch jobs on the same tables.

Streaming in AI systems

• Real-time features: “number of card transactions in the last 5 minutes” for a fraud model.
• Fresh retrieval: CDC or event streams update a RAG index within minutes of a source change.
• Application telemetry: every LLM call, retrieval result and user rating lands on a stream, then in the lake for evaluation.
• Live monitoring: detect input drift or a spike in refusals or errors as it happens.

Questions to ask

• What decision gets better with fresher data, and by how much?
• Is this computed on event time or processing time?
• How late can events arrive, and what happens to them?
• How big will the state grow, and when does it expire?
• If we deploy a bug, can we replay the log to fix the output?