Model-aware routing: different models on different boxes, one catalog to callers

Real installations run mixed hardware: a big-VRAM box with a large model, a few mid-range boxes with chat models, a CPU box for embeddings. In a model-aware farm the gateway routes each request to a worker that already has the requested model warm, and advertises the union of every worker's models as one catalog ^[1]. Callers see one model list and simply pick; placement is the gateway's problem, not theirs.

Warm beats cold by minutes

A model load can be a multi-gigabyte, minutes-long event. Landing a request on a box that already holds the model is the single biggest latency and efficiency win on heterogeneous farms ^[1]. Requests are routed warm-first: workers that report the requested model lead, and workers without it remain only as last-resort failover, where their own daemon would pull the model per its policy.

One catalog, many boxes

The gateway refreshes every worker's catalog on the health-check cadence, so the model list on the gateway is the deduplicated union of the whole farm, answered from cache ^[2]. A worker whose catalog cannot be fetched is treated as possibly warm rather than unavailable, so routing never degrades below the plain strategy.

Buy hardware per model class

Model-aware routing is always on in gateway mode, so the design lets you buy hardware per model class instead of making every box able to run everything ^[1]. Chat runs where the chat models live, embeddings on the CPU box, vision on the accelerator that suits it. Nothing extra is configured: build the farm, install each worker's intended models, and the gateway does placement.

Placement that keeps failover warm

Keep every latency-critical model on at least two workers so failover stays warm rather than triggering a cold load at the worst moment ^[2]. The load-balancing strategy still orders within the warm set, so least-connections or a weighted bias toward the big box composes cleanly with warmth.