Kafka Producer In-Flight Requests and Ordering

Kafka producers can send multiple requests without waiting for earlier acknowledgments. When requests fail and are retried, allowing several requests in flight can permit later records to be acknowledged before earlier records, potentially changing their order unless appropriate producer settings are used.

`max.in.flight.requests.per.connection` limits how many unacknowledged produce requests a producer can have outstanding on one connection to a broker. A higher value can improve throughput because the producer does not need to wait for each response before sending more data.

The ordering problem appears when an earlier request fails while a later request succeeds. With retries enabled, the producer may resend the failed request after the later request has already been processed. Without idempotence, this can result in records being written in a different order from the original send sequence.

Kafka's idempotent producer changes the situation. With idempotence enabled, Kafka requires compatible acknowledgement and retry settings and limits the in-flight request count to a maximum of 5. This allows multiple requests for throughput while the broker uses producer sequence numbers to detect duplicates and preserve the intended ordering for a partition.

Reducing the in-flight limit to `1` is a simple way to avoid overlapping producer requests, but it can reduce throughput. In practice, the right setting depends on whether ordering, throughput, idempotence, and failure behavior are more important for the workload.

The setting applies per producer-to-broker connection, not as one global queue across every broker. Since records belonging to a Kafka partition are handled by its leader, ordering discussions should always specify the partition being considered.

Key points

• `max.in.flight.requests.per.connection` controls outstanding unacknowledged requests per broker connection.
• Retries can cause later requests to complete before an earlier failed request is retried.
• With idempotence disabled, multiple in-flight requests plus retries can cause reordering.
• Kafka's idempotent producer supports up to 5 in-flight requests per connection while maintaining ordering semantics for a partition.
• Setting the limit to `1` serializes requests but can reduce throughput.

Interview questions

Why can multiple in-flight producer requests cause message reordering?

Suppose request 1 contains an earlier record and request 2 contains a later record. If request 1 fails after request 2 succeeds, the producer may retry request 1 after the later record has already been accepted. Without the protections of idempotence, the broker can therefore observe the later record before the retried earlier record. This is why retries and multiple outstanding requests must be considered together when discussing Kafka ordering.

What does `max.in.flight.requests.per.connection` control?

It limits the number of unacknowledged requests that a producer can have outstanding on a single broker connection. A higher value permits more concurrency and can improve throughput. However, more concurrent requests create more opportunities for failure and retry interactions to affect ordering. It is therefore a throughput-versus-ordering-risk setting rather than a simple batch-size setting.

How does idempotence affect the in-flight request limit?

Kafka's idempotent producer uses producer identity and sequence numbers to recognize duplicate or out-of-order retries. Kafka permits multiple outstanding requests with idempotence, but the compatible maximum for `max.in.flight.requests.per.connection` is 5. Idempotence also requires compatible producer settings such as acknowledgments and retries. This allows concurrency without giving up the producer's ordering and duplicate-prevention guarantees for a partition.

Would you set `max.in.flight.requests.per.connection` to 1 for strict ordering?

Setting it to 1 removes overlap between produce requests on a connection, which simplifies ordering behavior and can avoid the retry reordering scenario associated with multiple in-flight requests. However, it also reduces concurrency and may hurt throughput. With an idempotent producer, Kafka supports up to five in-flight requests while maintaining the producer's ordering guarantees for a partition. The choice should therefore be based on the required semantics and performance characteristics rather than treating 1 as universally necessary.

Is Kafka ordering guaranteed across an entire topic?

Kafka ordering is defined within a partition, not across all partitions of a topic. A producer can send records to different partitions, and those partitions are processed independently. The in-flight request discussion matters when considering the sequence of records assigned to the same partition and its leader. If an application requires a single total order, its records generally need to be routed to the same partition.