Sticky caller affinity: each user keeps their own warm worker

The <code>sticky</code> routing strategy maps every caller deterministically to a preferred worker, hashing the caller's install identity, then API key, then IP, in that order ^{<a href="#ref-1">[1]</a>}. Their requests keep landing on the same box, where their model is loaded and their prompt cache is hot. It is one field in the gateway configuration; the rest of the farm is unchanged.

Prompt-cache locality keeps chats fast

LLM runtimes reuse computation across a conversation, so bouncing a user between boxes discards that work and each turn pays again ^{<a href="#ref-1">[1]</a>}. Sticky affinity keeps multi-turn chats on the same warm lane, which is why it suits assistants and copilots with a stable user population. Stateless batch traffic, such as an embeddings sweep, is better spread by <code>least-connections</code>.

A hash, not a session table

The mapping is a hash, not stored session state, so there is nothing to replicate: it survives a gateway restart and multiple gateway replicas agree on it without coordination ^{<a href="#ref-2">[2]</a>}. Failover is deterministic per caller too, so a user's second-choice worker is always the same box and even failures behave consistently.

Adding a box remaps almost nobody

Because the strategy uses rendezvous hashing, adding or removing a worker only remaps the callers whose top choice changed; everyone else keeps their lane ^{<a href="#ref-1">[1]</a>}. A caller with no identity at all degrades to round-robin, and affinity composes automatically with model-aware routing by applying within the warm set of workers.