Kafka, from first principles

Apache Kafka is a durable, replayable log that many programs can write to and read from at their own pace. It's how Prayog gets every trade from the exchange to the post-trade, leaderboard and market-data services without slowing the exchange down. This page assumes no prior Kafka knowledge.

1. The problem

When a trade happens, several other parts of Prayog care: post-trade updates positions and P&L, the leaderboard re-ranks, the web terminal shows the fill. The exchange could call each of them directly, but then:

• a slow or crashed service would slow or block the exchange;
• adding a new consumer means changing the exchange;
• a service that was down for five minutes would have missed those trades forever.

Kafka sits in between. The exchange writes each event once and moves on. Each service reads at its own speed, and a service that was down simply continues from where it stopped.

2. A business example: the numbered order register

A wholesale market keeps a big bound register at the front desk. Every sale is written on the next numbered line, in ink. Nobody tears out pages.

• Accounts reads the register to bill customers; it keeps a bookmark at the last line it processed.
• Warehouse reads the same register to ship goods, with its own bookmark.
• If Accounts is out sick for a day, nothing is lost: tomorrow it continues from its bookmark.
• When the register gets too busy for one clerk, the market keeps three registers, one per product range. Each register is still strictly in order; across registers there is no single order.

The register is a topic, the three registers are partitions, line numbers are offsets, and bookmarks are committed offsets.

3. Topics, partitions and offsets

• Record: a key, a value (bytes: JSON in Prayog), headers and a timestamp.
• Topic: a named stream of records, like a table name.
• Partition: a topic is split into partitions, each an ordered, append-only log stored on disk. Partitions are the unit of parallelism and of ordering.
• Offset: a record's position in its partition. It never changes.
• Retention: records stay for a set time or size (7 days by default), whether or not anyone has read them. That's what makes replay possible.

4. Producing: keys decide the partition

The producer picks the partition from the record's key: hash(key) mod partitions. Same key, same partition, so records with the same key stay in order. Records with different keys may land in different partitions and be read in any relative order.

Prayog keys exchange events by symbol. All INFY events are in order relative to each other; INFY and TCS events can interleave. That's exactly the guarantee consumers need: an INFY position is built from INFY trades in order.

5. Consumer groups

Consumers that share a group.id form a consumer group. Kafka gives each partition to exactly one member of the group, so the members split the work:

• 3 partitions, 1 consumer: it reads all three.
• 3 partitions, 3 consumers: one partition each.
• 3 partitions, 4 consumers: one consumer sits idle. Partitions cap a group's parallelism; that's why the partition count is chosen up front.

When a member joins, leaves or crashes, the group rebalances: partitions are reassigned. The new owner starts from the group's committed offset for that partition, the bookmark, stored in Kafka's internal topic __consumer_offsets.

A different group (say, the leaderboard) reads the same records independently, with its own bookmarks. That's how one topic feeds many services.

6. Delivery guarantees

The question is when the consumer commits its bookmark, relative to doing the work:

Prayog uses at-least-once with idempotent consumers (BUILD_PLAN 16.2 #18). Every event carries eventId = the exchange's event sequence number. The plan for post-trade (S16): store the last applied id per partition in the same database transaction as the position update, and skip anything at or below it. A redelivered trade is recognised and ignored, so P&L is never counted twice.

7. Try it: produce, consume, crash

Three partitions, keyed by symbol. Produce some trades, set how many consumers the group has, and let them work. Then crash one between "processed" and "committed" and watch the redelivery.

8. Brokers, replicas and KRaft

• Broker: a Kafka server. A cluster has several; partitions are spread across them.
• Replication: each partition has a leader replica and followers on other brokers. Producers and consumers talk to the leader; followers copy it. If the leader's broker dies, an in-sync follower becomes leader. Production clusters use replication factor 3.
• Controller: keeps the cluster's metadata (which topics exist, who leads each partition). Kafka used to rely on a separate ZooKeeper cluster for this. KRaft (Kafka Raft) moved it into Kafka itself: a small quorum of controller nodes agree on metadata using the Raft consensus algorithm. Kafka 4.0 removed ZooKeeper entirely.

Prayog runs one node that is both broker and controller (KAFKA_PROCESS_ROLES=broker,controller), with replication factor 1. There's no redundancy, which is fine for a laptop, and the 3 partitions still make consumers deal with real partitioning.

9. Listeners: the most common setup mistake

A client connects in two steps:

1. It connects to a bootstrap address and asks for metadata: "which broker leads which partition, and how do I reach it?"
2. The broker answers with its advertised address, and the client opens new connections to that address for all real traffic.

So the advertised address must be reachable from wherever the client runs. Containers reach the broker as kafka; your Mac can't resolve that name, but can reach localhost:9094. Prayog therefore has one listener per network:

10. Kafka in Prayog

11. Questions

Why not put Kafka on the order path instead of the journal?

A network round trip to a broker (and its replicas, with acks=all) costs far more than an append to a local file, and Prayog's invariants forbid network calls on the order path. The journal makes the order durable; Kafka distributes the results afterwards.

Kafka or RabbitMQ?

RabbitMQ is a message broker: messages are routed to queues and removed once acknowledged. Kafka is a log: records stay for the retention period and any number of groups can read or re-read them. For "every service needs every trade, and must be able to catch up or replay", the log model fits.

Can I change the number of partitions later?

You can add partitions, but hash(key) mod partitions then sends some keys to a different partition, breaking per-key order across the change. Choose the count up front; that's why Prayog fixes it at 3 in the init job.