Capacity caps and honest backpressure: protecting small GPU boxes
A GPU box asked to run forty generations at once does not run them slowly; it thrashes or runs out of memory. A per-worker cap keeps each box in its sweet spot and fails fast when the pool is full.
A per-worker concurrency limit tells the gateway never to route more than a set number of in-flight requests to a box [1]. A saturated worker is skipped, and a fully saturated pool answers 503 with a Retry-After header immediately. That is honest backpressure rather than a queue that grows silently until something falls over.
Keep each box in its sweet spot
A box asked to run far more concurrent generations than it can handle does not degrade gracefully; it thrashes, runs out of memory, or takes its neighbours down with it [1]. A cap holds each box at the concurrency where its latency is still acceptable. On a mixed fleet you set a higher cap on the big box and a lower one on a small box, and each contributes what it actually can.
Fast-fail beats slow-death
When the pool is full, callers are told to retry shortly rather than left hanging until a timeout [1]. AI Suite apps honour Retry-After automatically and fall back to on-device AI where it is available, so a busy morning degrades into a brief wait instead of a stall [2].
Sizing the cap
Choose a cap by starting at the number of concurrent generations the box handles at acceptable latency and watching the worker card's latency on the dashboard as load rises; the knee is obvious [2]. A cap is hard: a saturated worker gets nothing, not even failover traffic, which is what protects it from being pushed past the edge at the worst moment. A cap of zero means uncapped.
References
- Software Tailor. "AI Server — product page." softwaretailor.com/ai-server.htm. Accessed 2026-07-22.
- Software Tailor. "AI Server documentation — Operations." softwaretailor.com/docs/ai-server/operations.htm. Accessed 2026-07-22.
- Microsoft. "AI Server on the Microsoft Store." apps.microsoft.com/detail/9P42956WBWCL. Accessed 2026-07-22.
Related articles
Stay healthy on the busiest day.
Per-box caps and honest backpressure keep a farm stable under load. Get it on Windows from the Microsoft Store, or read the feature tour first.