AI Server docs / Gateway & operations

Gateway & operations

Running a farm day to day: how the gateway routes, how to roll out safely, and how to watch all of it.

Gateway mode in one paragraph

Start the daemon with AISUITE_ENGINE=gateway (or --engine gateway) and it becomes a transparent front-end for a pool of worker AI Servers. Clients see a single AI Server — same endpoints, same keys, same behaviour; the gateway forwards each request byte-for-byte to a worker it chooses by health, load and model availability. The pool is defined in gateway.json in the gateway's data directory (workers, bearer tokens, strategy, options); the AI Server desktop app includes an editor for it (Server → Advanced → Multi-host gateway), or edit the file directly.

Routing

• Strategies: round-robin (default), least-latency (moving average), least-connections (current in-flight), weighted (per-worker weights), sticky (a given client consistently lands on the same worker).
• Model-aware: the gateway tracks each worker's installed models and routes a request first to a worker that already has the requested model warm — cold pulls only happen when nothing warm is available.
• Failover: transient worker failures (connection refused, 502/503/504) retry on another worker before the client ever notices; client errors (4xx) pass through verbatim, once.
• Backpressure: each worker carries an in-flight cap; when every worker is saturated the gateway answers 503 with Retry-After instead of queueing into a timeout. Shipped AI Suite clients honour it automatically.
• Attribution: every proxied response carries X-AISuite-Backend naming the serving worker; usage and audit records include it too.

Safe rollouts

• Zero-downtime upgrades: workers flip /readyz to 503 on SIGTERM, the gateway routes around them while in-flight requests finish, then they exit. Upgrade one worker at a time (or let Kubernetes rolling updates do it) and no request is dropped.
• Canary: mark a worker "canary": true and set canary_percent — a consistent slice of callers lands on canary workers (the same caller always gets the same side, so sessions don't flap). Watch the dashboard's per-worker error/latency split, then promote or roll back.
• Model warm-up: add a new worker, let it pull the models it needs, and the model-aware router shifts traffic to it as its catalog fills — no manual cutover.

Watching it: dashboard

Every node (worker or gateway) self-hosts GET /dashboard — a self-contained live page showing server state (ready/draining), request rates, latency percentiles, top endpoints, and on a gateway the full farm view: each worker's health with a plain-language reason, canary badges, in-flight load against its cap, latency, and installed models, plus usage by app, worker and model. The dashboard shell is anonymous and carries zero data itself (the aggregate JSON behind it requires an API key off-loopback). Set AISUITE_DASHBOARD=0 to unmap it entirely.

Watching it: Prometheus

GET /v1/server/metrics serves native Prometheus text exposition — no exporter or sidecar. It is API-key authenticated: give your scraper its own key. Edge series cover requests, errors, per-endpoint counts and latency summaries; a gateway additionally exposes pool-level and per-worker series (health, in-flight, saturation) — the signals to alert on for "N workers healthy" and to feed an autoscaler (see Kubernetes & Helm).

scrape_configs:
  - job_name: aisuite
    metrics_path: /v1/server/metrics
    authorization: { type: Bearer, credentials: <scraper-api-key> }
    static_configs: [{ targets: ["gateway.aisuite.svc:8080"] }]

Usage accounting

Usage is recorded per app, per model, per API key and (behind a gateway) per worker — request counts and token counts, with no prompt or response content, ever. Query it from the Usage page in the desktop app, the dashboard, or GET /v1/server/usage (e.g. ?by=worker). That's the FinOps view: which apps and models consume which hardware.

Security hardening checklist

• One API key per caller (they're cheap; revocation is then surgical). Only key hashes are stored on disk.
• TLS: terminate at your ingress/proxy in front of the gateway and set AISUITE_TRUST_PROXY=1 there — and only there. Never set it on a node directly reachable by clients.
• Restrict worker reachability to the gateway (NetworkPolicy or firewall) — workers hold their own keys and reject unauthenticated calls regardless, but defense in depth is free here.
• Keep the pool file (gateway.json) in secret storage — it contains worker bearer tokens.
• Audit export is signed; store bundles somewhere append-only if your compliance regime needs tamper-evidence.
• Rate limits, quotas and budgets are per-key governance settings (Commercial tier) — set a budget on every key you hand to a team you don't control.

Next: API & clients · Licensing & activation