Skip to content

Kafka

On the CV twice: the work stack and Prayog's publisher + ledger. Expect both concept questions and "how did you use it".

Architecture

flowchart LR
  P1[Producer] -->|key = account| T
  subgraph T[Topic: trades, 3 partitions]
    p0[Partition 0<br/>offsets 0..n]
    p1[Partition 1]
    p2[Partition 2]
  end
  p0 --> C1
  p1 --> C1
  p2 --> C2
  subgraph G[Consumer group: post-trade]
    C1[Consumer 1]
    C2[Consumer 2]
  end
Explain Kafka's core concepts.
  • Broker: a Kafka server. A cluster is many brokers.
  • Topic: a named stream of records, split into partitions.
  • Partition: an ordered, append-only log; each record gets an increasing offset. The unit of parallelism and ordering.
  • Replication: each partition has a leader and followers on other brokers; the ISR (in-sync replicas) are followers caught up with the leader.
  • Producer: writes records (key, value, headers) to a topic; the key picks the partition (hash of key).
  • Consumer group: consumers sharing a group.id split the partitions; each partition is read by exactly one consumer in the group. Different groups each get all the data.
  • Offset commit: the group stores its position per partition (in the __consumer_offsets topic).
  • KRaft: Kafka's built-in Raft metadata quorum, replacing ZooKeeper (ZooKeeper removed in Kafka 4.0).
How does Kafka guarantee ordering?

Only within a partition. Records with the same key go to the same partition, so they stay in order. To keep all events of an account in order, key by account id. No ordering across partitions. Producer retries could reorder unless the producer is idempotent (enable.idempotence=true, default since Kafka 3.0), which keeps order with up to 5 in-flight requests.

How many partitions should a topic have?

Partitions cap consumer parallelism (max useful consumers in a group = number of partitions). Size from target throughput ÷ per-consumer throughput, with headroom; you can add partitions later but that changes key → partition mapping (breaks ordering for existing keys), so start with enough. Too many costs broker memory, file handles and longer failover.

Why is Kafka fast?

Sequential disk writes to an append-only log, the OS page cache instead of an in-process cache, zero-copy (sendfile) from page cache to socket, batching and compression of records, and partitions spread load across brokers.

Delivery guarantees

At-most-once vs. at-least-once vs. exactly-once?
  • At-most-once: commit offset before processing; a crash loses messages.
  • At-least-once: process, then commit; a crash after processing but before commit reprocesses messages (duplicates). The common default; make consumers idempotent.
  • Exactly-once: Kafka transactions (transactional.id) with idempotent producers, for read-process-write within Kafka (consume, produce, commit offsets atomically; consumers use isolation.level=read_committed). Writing to an external database needs idempotency or storing offsets in the same DB transaction.
What do acks=0/1/all and min.insync.replicas mean?

acks=0: don't wait (may lose). acks=1: leader wrote it (lost if leader dies before followers copy). acks=all: all ISR replicas have it. With replication.factor=3 and min.insync.replicas=2, a write succeeds only when at least 2 replicas have it, so one broker can fail without data loss; if ISR drops below 2, producers get NotEnoughReplicas instead of silently losing durability.

What is an idempotent producer?

The broker gives the producer an id and tracks a sequence number per partition, so a retried batch is written once and in order. Prevents duplicates from producer retries (not from your application sending twice).

How do you make a consumer idempotent?

Give each event a unique id, and record processed ids in the same database transaction as the effect (or use an upsert keyed by the id), so reprocessing is a no-op. Or design operations to be naturally idempotent ("set balance to X" rather than "add Y").

Prayog link: the post-trade ledger applies each batch in one Postgres transaction and skips event ids already applied, so a redelivered batch changes nothing. The ledger is checked as zero-sum after drills where Kafka or post-trade is killed mid-session.

Consumers in practice

What is a rebalance? What triggers it and why is it a problem?

The group coordinator reassigns partitions when a consumer joins, leaves, crashes (misses heartbeats beyond session.timeout.ms) or takes too long between polls (max.poll.interval.ms). With the old eager protocol every consumer stops ("stop the world"). Mitigate with the cooperative sticky assignor (incremental), static membership (group.instance.id) for rolling restarts, and keeping per-poll processing short (max.poll.records). Kafka 4.0's new consumer group protocol (KIP-848) moves assignment to the broker and makes rebalances incremental.

Consumer lag: what is it and how do you handle growing lag?

Lag = latest offset − committed offset per partition. Growing lag means consumers can't keep up. Check: is processing slow (DB, external calls) or stuck (poison message, rebalancing loop)? Fix: scale consumers up to the partition count, batch DB writes, process asynchronously with care for ordering, add partitions if needed. Alert on lag.

Prayog link: prayog_kafka_lag in Grafana; during the Kafka-down drill it climbs and then drains.

How do you handle a poison message (one that always fails)?

Retry a bounded number of times with backoff, then send it to a dead-letter topic with the error in headers, commit, and continue; alert on the DLT and replay after fixing. Never block a partition forever. Spring Kafka: DefaultErrorHandler + DeadLetterPublishingRecoverer; non-blocking retries with @RetryableTopic.

Auto-commit vs. manual commit?

enable.auto.commit=true commits offsets periodically from poll(), which can commit before processing finishes (loss on crash) or after (duplicates). Manual commit after processing gives at-least-once. Spring Kafka's container commits after the listener returns (AckMode.BATCH default) or on Acknowledgment.acknowledge().

Storage and topics

Retention vs. log compaction?

Retention deletes old segments by time (retention.ms, default 7 days) or size. Compaction keeps only the latest record per key (plus tombstones for deletes): a topic becomes a changelog/table of current state (e.g. latest price per symbol, user settings).

How does Kafka handle broker failure?

The controller elects a new leader for each partition from the ISR. Producers and consumers refresh metadata and reconnect. With unclean.leader.election.enable=false (default), an out-of-sync replica is never elected, so no data loss but the partition may be unavailable until an ISR member returns.

Comparisons and design

Kafka vs. RabbitMQ vs. SQS?

Kafka: a durable log; consumers pull and track offsets; replay any time; very high throughput; ordering per partition. RabbitMQ: a message broker with flexible routing (exchanges, queues); messages are removed once acknowledged; per-message acks, priorities; good for task queues. SQS: managed queue, no ops; standard (at least once, best-effort order) or FIFO. Choose Kafka for event streams, many consumers, replay; a queue for work distribution.

Why did Prayog keep Kafka off the order path?

The order is answered once it's in the local journal (fsync). A separate publisher tails the event log and sends to Kafka with a file checkpoint of the last contiguous acked event. If Kafka is down, trading continues and events wait on disk; when Kafka returns, the publisher resends from the checkpoint. Duplicates are possible, gaps are not; consumers dedupe by event id. Same principle as the transactional outbox: the durable record first, the event stream derived from it.

A bug in Prayog's publisher: tell me about it.

A late ack from an old batch could mark a dead connection healthy: the error callback set a failure flag, but a later success callback from a replaced producer set connected back to true. Fix: a failed send reports disconnected at once, and callbacks from a replaced producer are ignored. Lesson: callbacks from an old generation of a resource must not change the new one's state (failure 3).

How would you design an event schema and evolve it?

Use a schema (Avro/Protobuf/JSON Schema) with a registry; include an event id, type, version, timestamp and the key; make changes backward compatible (add optional fields with defaults, never repurpose or remove a field consumers use). Consumers ignore unknown fields.

Spring Kafka basics?

KafkaTemplate.send(topic, key, value) returns a future; @KafkaListener(topics=..., groupId=...) with a ConcurrentKafkaListenerContainerFactory (concurrency ≤ partitions); serializers/deserializers (JSON, Avro); ErrorHandlingDeserializer so a bad payload doesn't crash the consumer; @EmbeddedKafka or Testcontainers for tests.