Caching Trade-offs

Caching can lower read latency and backing-store load when data is reused, but it costs memory and adds consistency and operational complexity. Its value depends on workload, acceptable staleness, and how misses and failures are handled.

A cache keeps copies of data in a faster or closer storage layer so repeated reads can avoid a slower backing store such as a database or remote service. In cache-aside, the application checks the cache first; on a miss, it reads the backing store and populates the cache. This can reduce response time and backend work when requests reuse data.

The benefit depends on workload. Frequently reused values are good candidates; one-off or rapidly changing values may bring little improvement. A cache miss still pays the backing-store cost and incurs cache-management work, so average performance and tail latency depend on hit rate, miss behavior, and the cost of each layer.

Cached data can be stale after the source changes. Expiration limits how long an entry is retained, while invalidation tries to remove or update it when writes occur. Neither is free: short expiration raises miss traffic, and invalidation introduces coordination and failure cases. The application must decide what degree of staleness is acceptable for each data type.

Caches also consume memory and add operational complexity. Capacity limits, eviction policy, hot keys, cache outages, and bursts of simultaneous misses can affect reliability. A robust design treats the cache as an optimization when possible, bounds resource use, and plans how the service behaves when entries are absent or the cache is unavailable.

Key points

• Caching trades memory and operational complexity for lower latency and reduced load on slower dependencies.
• The hit rate and miss cost determine whether caching improves the workload.
• Expiration and invalidation manage staleness but introduce their own costs and failure modes.
• Plan for capacity limits, cache outages, and bursts of concurrent misses.

Interview questions

How would you prevent a cache stampede for a popular key?

A stampede happens when many requests see the same key as absent or expired and all fetch it from the source at once. This can overwhelm the backing service precisely when the cache is least helpful. Per-key request coalescing or a short-lived lock can let one request refresh the entry while others wait or use an acceptable stale value. Jittered expiration can also reduce synchronized expiry across many keys, though it does not by itself prevent concurrent refreshes for one hot key.

How does the cache-aside pattern work, and when would you use it?

Cache-aside is an application-managed pattern: read the cache first, and on a miss read the source and populate the cache. It is straightforward to add around existing data access and works well when reads reuse values. The application must still choose expiration and invalidation behavior, and handle misses and cache failures.

How do you choose a cache TTL?

A shorter TTL usually reduces the period in which a changed source value can remain cached, but causes more expirations and backing-store reads. A longer TTL can improve hit rate and reduce load, while allowing stale values to persist longer. The right choice depends on freshness requirements, update frequency, miss cost, and traffic. Measure hit rate, latency, and backend load, and use explicit invalidation where the freshness requirement warrants its added coordination.

What consistency problems can happen when updating data that is also cached?

With cache-aside, a write commonly updates the source and then invalidates the corresponding cache entry. If the process fails between those operations, the cache may continue serving old data until expiration or another repair mechanism acts. Updating the cache directly can avoid some stale reads but creates its own consistency and partial-failure questions. The appropriate strategy depends on the allowed staleness, write frequency, and whether the application can tolerate extra coordination.