Caching for API Performance

Caching reuses previously computed or retrieved results to reduce API latency and load on backing services. It improves performance only when key design, freshness rules, and failure behavior match the application's correctness requirements.

Caching stores a result so a later request can avoid repeating expensive work, such as a database query, remote API call, or computation. A cache is typically faster than the underlying source, but may have limited capacity and may not contain the latest value.

In a common cache-aside flow, the application looks up a key, returns a hit if present, and otherwise reads from the source, stores the result, and returns it. Keys should encode every input that affects the result, including relevant authorization or tenant context; otherwise one caller could receive another caller's data.

Cached values need a freshness policy. Time-to-live (TTL) bounds how long an entry remains usable, while explicit invalidation removes or refreshes entries when underlying data changes. Neither approach alone guarantees perfect freshness: expiration permits staleness until the TTL elapses, and invalidation can be missed or delayed.

Caching can reduce latency and load, but introduces consistency and failure-mode concerns. A popular key expiring for many requests at once can cause a cache stampede against the backing store; mitigations include request coalescing, staggered expiration, and stale-while-revalidate. Consider cache outages, memory limits, sensitive data, and metrics such as hit rate and backend load when designing a cache.

Key points

• Caching trades reduced repeated work and latency for storage use and possible staleness.
• Choose cache keys that uniquely represent all inputs affecting the returned value.
• Use TTLs, invalidation, or both to define how cached data becomes fresh.
• Plan for misses, cache failures, evictions, and concurrent refreshes.
• Measure application impact with latency, hit rate, and backing-store load.

Interview questions

How would you design cache keys for a multi-tenant API?

The key must distinguish all inputs that can change the result, such as resource ID, locale, tenant, and authorization scope when applicable. Omitting a result-affecting input can return incorrect data, and omitting tenant or access context can expose data across users. Keys should also be consistently serialized and versioned when the cached representation changes.

What is cache-aside, and what happens on a cache miss?

In cache-aside, the application checks the cache before querying the source of truth. On a miss, it reads from the source, places the result in the cache, and returns it. This is straightforward and lets the application choose what to cache, but correctness depends on key design and a freshness or invalidation policy.

What is a cache stampede, and how can you mitigate it?

A stampede occurs when many requests miss or expire the same popular key and concurrently query the backing service. Request coalescing can make concurrent callers wait for one in-flight load, while jittered TTLs reduce synchronized expirations across keys. Stale-while-revalidate can serve an older value temporarily while one request refreshes it, if the application can tolerate that staleness.

What tradeoffs does adding a cache introduce to an API?

A cache hit can improve latency and reduce load on the database or remote service, but cached data may be stale and the cache can fail independently. I would decide whether a request can fall back to the source, fail fast, or serve stale data, based on the operation's correctness requirements and the source's capacity. I would monitor hit rate, latency, eviction, errors, and backend load rather than treating a high hit rate as proof of correctness.