Autoscaling

Autoscaling automatically adjusts compute capacity in response to workload or resource signals. Effective policies meet service objectives while respecting reaction time, cost bounds, and the limits of downstream dependencies.

Autoscaling is the automatic adjustment of compute capacity as demand changes. A policy observes signals such as request rate, queue depth, CPU use, or latency, then changes the number or size of resources within configured limits.

Horizontal scaling adds or removes instances, containers, or workers. It is commonly used for stateless backend services because traffic can be spread across replicas. Vertical scaling changes the capacity of an existing resource, for example by selecting a larger machine; it has practical limits and may require a restart or replacement.

A useful policy needs more than a metric and a threshold. Minimum and maximum capacity bound cost and availability, while cooldowns, stabilization windows, or gradual changes can reduce oscillation. Scale-out and scale-in thresholds are often intentionally different so small fluctuations do not repeatedly reverse the action.

Autoscaling is not instantaneous. Provisioning, startup, health checks, and readiness take time, so capacity planning must account for the time between detecting demand and serving traffic. Keep baseline capacity for expected load, monitor leading signals such as queue growth, and ensure dependencies and data stores can tolerate the added concurrency.

For a Python backend, horizontal scaling works best when request handling is stateless or state is externalized. Per-process memory, local files, background jobs, connection pools, and database limits can make adding workers unsafe or ineffective unless their behavior is designed for multiple replicas.

Key points

• Autoscaling adjusts resource capacity from workload or resource signals.
• Horizontal scaling changes replica count; vertical scaling changes an individual resource's size.
• Use bounds, deliberate scale-in behavior, and stabilization to manage cost and oscillation.
• Scaling delay means baseline capacity and leading indicators still matter.
• Horizontal replicas require attention to process-local state and downstream concurrency.

Interview questions

What is autoscaling, and what problem does it solve?

Autoscaling adjusts capacity automatically based on observed demand or resource metrics and a defined policy. It can add or remove replicas horizontally, or adjust resource size vertically depending on the platform and configuration. The goal is to meet service objectives while avoiding unnecessary idle capacity. Bounds and safe scale-in behavior are important because a metric threshold alone does not capture startup delay, dependency limits, or disruption risk.

How would you design autoscaling for a Python HTTP API?

For a stateless HTTP service, I would usually scale replica count using a signal that tracks demand or saturation, such as requests per instance, CPU, or latency. I would configure minimum and maximum replicas, readiness and health checks, and a scale-out response fast enough for the service's startup time. Scale-in should be more conservative, allowing requests to drain and avoiding rapid reversals. I would verify that the load balancer, database, and other dependencies can handle the resulting concurrency.

What are the main failure modes in an autoscaling design?

I would first identify the actual bottleneck and choose a metric with a meaningful relationship to unmet work or the service objective. Then I would measure provisioning delay, define a target operating range, set bounds, and use stabilization or asymmetric scale-out and scale-in behavior to reduce oscillation. I would model the maximum concurrency the database and external dependencies can support, and cap replicas accordingly or introduce admission control. Finally, I would validate behavior with load tests and production telemetry, including scale events, saturation, errors, and latency during transitions.

How would you autoscale background workers in a Python service?

A CPU-based policy is reasonable when CPU saturation tracks the workload that limits throughput, but it can miss queue buildup caused by I/O waits or downstream slowness. A queue-backed worker is often better scaled using queue depth or estimated time to drain, perhaps normalized by active workers. The target should reflect acceptable processing delay, and scale-in must not remove workers that are executing non-interruptible jobs. I would also cap workers to protect shared dependencies and account for job duration and startup time.