Log Segmentation

Append-only logs are divided into segments so retention can remove complete inactive segments rather than deleting individual records. Segments also bound the amount of data that must be managed for indexing, recovery, and storage operations.

A log segment is a contiguous portion of an append-only log stored as its own physical unit. Producers normally append to the active segment until a rollover condition is reached, such as a size or time threshold, after which a new segment becomes active.

Segmentation makes retention efficient because expiration can operate on whole inactive segments. Instead of scanning and deleting individual records throughout a large log, the system can remove segments whose records are old enough according to its retention policy.

The active segment is generally treated differently from closed segments because new records are still being appended to it. Consequently, retention cleanup usually waits until data is represented by an eligible inactive segment. This means the amount of data retained can temporarily exceed the nominal retention duration or size.

Segment size and rollover frequency create a trade-off. Smaller segments allow retention to discard expired data more precisely and can reduce the amount of excess data kept, but they increase the number of files and associated metadata operations. Larger segments reduce segment-management overhead but can make cleanup coarser and delay removal of expired records.

Key points

• Logs are split into independently managed segments.
• Writes normally target one active segment at a time.
• Rollover creates a new segment when size or time conditions are met.
• Retention can delete complete inactive segments efficiently.
• Smaller segments improve cleanup granularity but increase management overhead.

Interview questions

How does segment size affect retention behavior?

Smaller segments give retention more granular deletion boundaries because expired data is more likely to be isolated in an eligible segment. However, they increase the number of segments and the associated metadata and file-management work. Larger segments reduce that overhead but can cause more expired data to remain until an entire segment becomes eligible.

How are log segmentation and partitioning different?

Partitioning creates separate logical streams, each with its own ordering and storage path. Segmentation subdivides the stored history within one such stream into sequential chunks. A single partition can therefore contain many segments over its lifetime.

Why do append-only logs use segments for retention?

Segments let the storage system treat a group of records as one management unit. When a segment is no longer needed, the system can remove the whole segment rather than locating and deleting every expired record. This makes retention much cheaper for large append-only logs.

What causes a log segment to roll?

A segment rolls when the system reaches a configured rollover condition. Common conditions include the segment reaching a size threshold or exceeding a time threshold. The current segment is closed for further appends and a new segment becomes active.

What trade-offs would you consider when choosing segment size?

I would balance retention granularity, storage-management overhead, recovery behavior, and workload characteristics. Smaller segments can make cleanup more precise, but they increase the number of files and metadata operations. Larger segments reduce that overhead but can delay removal of expired data and make individual storage units larger to process.