Log Compaction in Kafka

Log compaction retains the newest record for each key, allowing older records for that key to be removed. It is useful when a topic represents the latest state of keyed entities rather than a complete history of every event.

In Apache Kafka, log compaction is a retention policy applied to a topic partition. Kafka periodically rewrites older portions of the log so that, for a given key, the most recent record is retained while superseded records can eventually be discarded.

Compaction is key-based, not simply time-based. If a key appears multiple times, earlier values become obsolete once a newer value for that key exists. Records with different keys remain independently eligible for compaction.

A tombstone is a record whose value is `null`. For a compacted topic, a tombstone marks a key as deleted. Kafka retains the tombstone for a configured period before it can also be removed, allowing consumers that are catching up to observe the deletion.

Compaction does not mean that the partition immediately contains exactly one record per key. Compaction runs asynchronously, and duplicate records can temporarily remain. Kafka also preserves ordering within each partition, while the compacted log represents the latest known value for retained keys rather than an immutable event history.

Key points

• Compaction keeps the latest record for each key rather than retaining every historical value indefinitely.
• Compaction is asynchronous, so duplicate records can exist until a cleaner processes them.
• A `null` value acts as a tombstone that represents deletion of a key.
• Compacted topics are appropriate for state snapshots, caches, and changelog-style data.
• Compaction is different from ordinary time-based retention; a compacted topic can retain old data based on keys rather than only age.

Interview questions

What is Kafka log compaction, and why would you use it?

Log compaction is a Kafka retention mechanism that keeps the newest record for each key while allowing older records with that key to be removed. It is useful when consumers primarily need the current state of keyed data rather than every historical update. Examples include user profiles, configuration, and database changelogs. Compaction runs in the background, so old records may remain for some time.

How does Kafka represent deletion in a compacted topic?

Kafka uses a tombstone: a record with a key and a `null` value. The tombstone tells consumers that the key should be considered deleted. Kafka retains the tombstone for a configured period so consumers that are behind can observe the deletion. After that period, the tombstone itself may be removed during compaction.

Does log compaction guarantee that only one record per key remains?

No. Compaction is asynchronous, so multiple records for the same key can coexist while a partition is waiting to be cleaned. Kafka also does not treat compaction as an operation that instantly rewrites the entire partition. Once the relevant log segments are cleaned, obsolete records can be discarded. Therefore consumers should not depend on physical uniqueness of keys at every moment.

How would you choose between compacted and time-retained Kafka topics?

The choice depends on whether historical versions are required. A compacted topic is appropriate when consumers need the latest value for each key and old versions are usually redundant. Time-based retention is more appropriate when every event should remain available for a defined period. Kafka topics can also be configured with policies that combine compaction with deletion, depending on the required lifecycle.

What are important design considerations when using compacted topics for a database changelog?

The key must identify the entity whose latest state should be retained, and deletion must be represented consistently with tombstones. Consumers should tolerate duplicate records and should not assume compaction has already occurred. The design must also account for ordering within partitions, because records for a given key are ordered only within their partition. Finally, retention and recovery requirements should determine whether compaction alone is sufficient or whether historical events must be preserved elsewhere.