Sticky caller affinity: each user keeps their own warm worker
Bouncing a user between workers throws away the prompt cache their conversation built. Sticky affinity keeps each caller on one warm box, using a hash rather than a session table.
The sticky routing strategy maps every caller deterministically to a preferred worker, hashing the caller's install identity, then API key, then IP, in that order [1]. Their requests keep landing on the same box, where their model is loaded and their prompt cache is hot. It is one field in the gateway configuration; the rest of the farm is unchanged.
Prompt-cache locality keeps chats fast
LLM runtimes reuse computation across a conversation, so bouncing a user between boxes discards that work and each turn pays again [1]. Sticky affinity keeps multi-turn chats on the same warm lane, which is why it suits assistants and copilots with a stable user population. Stateless batch traffic, such as an embeddings sweep, is better spread by least-connections.
A hash, not a session table
The mapping is a hash, not stored session state, so there is nothing to replicate: it survives a gateway restart and multiple gateway replicas agree on it without coordination [2]. Failover is deterministic per caller too, so a user's second-choice worker is always the same box and even failures behave consistently.
Adding a box remaps almost nobody
Because the strategy uses rendezvous hashing, adding or removing a worker only remaps the callers whose top choice changed; everyone else keeps their lane [1]. A caller with no identity at all degrades to round-robin, and affinity composes automatically with model-aware routing by applying within the warm set of workers.
References
- Software Tailor. "AI Server — product page." softwaretailor.com/ai-server.htm. Accessed 2026-07-22.
- Software Tailor. "AI Server documentation — Operations." softwaretailor.com/docs/ai-server/operations.htm. Accessed 2026-07-22.
- Microsoft. "AI Server on the Microsoft Store." apps.microsoft.com/detail/9P42956WBWCL. Accessed 2026-07-22.
Related articles
Give every user a warm lane.
Sticky affinity keeps multi-turn chats fast across a farm. Get it on Windows from the Microsoft Store, or read the feature tour first.