Model-aware routing: different models on different boxes, one catalog to callers
On mixed hardware, landing a request on a box that already has the model loaded is the biggest efficiency win there is. The gateway routes warm-first and shows callers one catalog.
Real installations run mixed hardware: a big-VRAM box with a large model, a few mid-range boxes with chat models, a CPU box for embeddings. In a model-aware farm the gateway routes each request to a worker that already has the requested model warm, and advertises the union of every worker's models as one catalog [1]. Callers see one model list and simply pick; placement is the gateway's problem, not theirs.
Warm beats cold by minutes
A model load can be a multi-gigabyte, minutes-long event. Landing a request on a box that already holds the model is the single biggest latency and efficiency win on heterogeneous farms [1]. Requests are routed warm-first: workers that report the requested model lead, and workers without it remain only as last-resort failover, where their own daemon would pull the model per its policy.
One catalog, many boxes
The gateway refreshes every worker's catalog on the health-check cadence, so the model list on the gateway is the deduplicated union of the whole farm, answered from cache [2]. A worker whose catalog cannot be fetched is treated as possibly warm rather than unavailable, so routing never degrades below the plain strategy.
Buy hardware per model class
Model-aware routing is always on in gateway mode, so the design lets you buy hardware per model class instead of making every box able to run everything [1]. Chat runs where the chat models live, embeddings on the CPU box, vision on the accelerator that suits it. Nothing extra is configured: build the farm, install each worker's intended models, and the gateway does placement.
Placement that keeps failover warm
Keep every latency-critical model on at least two workers so failover stays warm rather than triggering a cold load at the worst moment [2]. The load-balancing strategy still orders within the warm set, so least-connections or a weighted bias toward the big box composes cleanly with warmth.
References
- Software Tailor. "AI Server — product page." softwaretailor.com/ai-server.htm. Accessed 2026-07-22.
- Software Tailor. "AI Server documentation — Kubernetes." softwaretailor.com/docs/ai-server/kubernetes.htm. Accessed 2026-07-22.
- Microsoft. "AI Server on the Microsoft Store." apps.microsoft.com/detail/9P42956WBWCL. Accessed 2026-07-22.
Related articles
Use the hardware you have, intelligently.
Different models on different boxes, one catalog to callers. Get it on Windows from the Microsoft Store, or read the feature tour first.