Canary rollouts: trial a new model on a slice of real traffic
Before you move a fleet onto a new model, try it on a controlled slice of real traffic. Canary routing gives a consistent percentage of callers the new build and keeps the rest on stable.
Flag one or more workers as canary and give the pool a canary percentage: that share of callers, assigned consistently by an identity hash rather than per-request coin flips, prefer the canary worker while everyone else stays on stable [1]. Both groups remain each other's failover, so the change is reversible and contained.
Try new models on real traffic
New model versions, new quantizations, and new daemon builds can be trialled on a controlled slice of real traffic before the fleet commits to them [1]. This is how you run an A/B-style comparison of a new model against the incumbent using live requests rather than a synthetic bench that may not match your workload.
Coherent, not flapping
Caller-consistent slicing means a user is either fully on the trial or fully off it, so feedback is coherent and comparisons are meaningful with no flapping mid-conversation [1]. A user who is in the canary group stays there for their whole session, which is what makes the resulting comparison trustworthy.
Fail-safe by construction
If the canary box dies or misbehaves, its slice fails over to stable workers automatically [1]. The blast radius is the slice, and only until failover. Pair a canary with a drain-aware rolling upgrade, which is the mechanical rollout, and a fleet-wide model change becomes a graduated decision rather than a leap [2].
References
- Software Tailor. "AI Server — product page." softwaretailor.com/ai-server.htm. Accessed 2026-07-22.
- Software Tailor. "AI Server documentation — Operations." softwaretailor.com/docs/ai-server/operations.htm. Accessed 2026-07-22.
- Microsoft. "AI Server on the Microsoft Store." apps.microsoft.com/detail/9P42956WBWCL. Accessed 2026-07-22.
Related articles
Adopt new models without betting the office.
Trial a build on a controlled slice, then commit the fleet. Get it on Windows from the Microsoft Store, or read the feature tour first.