AI Gateway: one endpoint in front of a farm of private AI Servers
One URL and one key in front of a pool of worker AI Servers: health-checked load balancing, transparent failover, model-aware routing, and Kubernetes auto-discovery — with no vendor cloud in the request path.
A single AI Gateway sits in front of a pool of worker AI Servers. Applications connect to the gateway with one URL and one key, exactly as if it were a single AI Server; behind it, the gateway load-balances every request across the workers, health-checks each one, and moves a dying worker's traffic to a healthy one before the caller notices [1]. It is not a separate product. AI Gateway is AI Server running in its farm front-end mode, so the OpenAI-compatible /v1/* and Ollama-compatible /api/* surfaces a single server exposes are exactly what the farm exposes [1].
One URL and one key, a farm behind it
The gateway balances every request type the platform serves: chat, embeddings, image generation, vision, and audio, including the live-speech WebSocket [1]. Requests are placed by a strategy you choose — round-robin, least-latency, least-connections, weighted, or sticky per client. Model-aware routing sends a request to a worker that already has the requested model warm, which avoids a cold-load penalty when a colder machine could otherwise have taken the call [1].
Add a GPU box and capacity grows; clients change nothing. Take a box down for maintenance and its traffic drains gracefully, so no in-flight request is dropped. We kill workers mid-traffic in our own test suite to keep that guarantee honest: a worker that dies is retried on a healthy one, and the failover happens inside the gateway before the client sees an error [1][3].
The trust boundary a farm introduces
A farm changes who holds which credential, and the design keeps that boundary tight. The client's gateway key never reaches a worker. The gateway presents each worker its own key, so a leaked gateway key cannot be replayed straight against a worker, and per-worker credentials stay inside the farm [2]. Caller identity, meaning which app and which install issued a request, passes through to the worker. Each worker's content-free audit log therefore records the real originator rather than attributing everything to the gateway [2].
Backpressure instead of timeouts
When every worker is saturated, the gateway returns an explicit "retry shortly" rather than holding a connection open until it times out [1]. Callers get a signal they can act on. Two rollout controls ride the same routing layer. A canary release sends a consistent slice of traffic to new machines before they carry full load. A zero-downtime upgrade drains a worker, upgrades it, and returns it to the pool, with the endpoint staying up throughout [1].
Kubernetes without manual pool edits
On Kubernetes, workers are discovered automatically as they scale, so an autoscaling farm needs no manual edits to the gateway's worker list [2]. The recipe ships three shapes from one product: Docker Compose for a single-box demo, Kubernetes manifests, and a Helm chart for the cluster. The product, the APIs, and the license are identical from one laptop to a GPU farm [2].
What a farm costs you to operate
Every server, the gateway included, hosts its own live dashboard and exposes native Prometheus metrics for fleet health, latency, usage, and capacity [3]. Fleet visibility is one dashboard, not a monitoring stack assembled before the first request. Network serving stays gated at every node: a non-loopback bind fail-closes unless the node carries a Pro entitlement and at least one API key, which stops an unlicensed or unkeyed box from quietly exposing itself on the LAN [1][4].
A gateway farm is the topology behind the platform's availability position: the AI endpoint stays up when a box does not, and it does so without a vendor cloud in the request path. It is licensed as part of Pro Commercial rather than sold as a separate product [4]. Read the full feature tour on the AI Server product page.
References
- Software Tailor. "AI Server — product page." softwaretailor.com/ai-server.htm. Accessed 2026-07-22.
- Software Tailor. "AI Server documentation — Kubernetes." softwaretailor.com/docs/ai-server/kubernetes.htm. Accessed 2026-07-22.
- Software Tailor. "AI Server documentation — Operations." softwaretailor.com/docs/ai-server/operations.htm. Accessed 2026-07-22.
- Microsoft. "AI Server on the Microsoft Store." apps.microsoft.com/detail/9P42956WBWCL. Accessed 2026-07-22.
Related articles
Scale private AI without changing a client.
Start free on one machine; add workers behind a gateway when one box isn't enough. Get it on Windows from the Microsoft Store, or read the feature tour first.