AI Server docs / Kubernetes & Helm

Kubernetes & Helm

The production topology: an AI Gateway Deployment fronting a StatefulSet of workers, discovered by DNS, licensed by a Secret, upgraded with zero downtime.

Install with Helm (recommended)

helm repo add softwaretailor https://softwaretailor.com/charts
helm repo update

kubectl create namespace aisuite
kubectl -n aisuite create secret generic aisuite-license --from-literal=key=<your-license-key>
kubectl -n aisuite create secret generic aisuite-worker-keys --from-file=server-keys.json

helm install aisuite softwaretailor/aisuite -n aisuite \
  --set license.existingSecret=aisuite-license

The chart deliberately refuses to render configurations that would install cleanly and then not work — no license, no API keys, an autoscaler pointed at a static pool, CPU-based autoscaling on GPU workers, a real engine with no GPU. If helm install rejects your values, the message tells you which invariant you broke; that is the chart doing its job.

Kustomize users: the same topology ships as reference manifests (base + autoscale and gpu overlays) in the public deploy repository — kubectl apply -k after replacing the placeholder Secrets.

Why the topology looks the way it does

• Workers are a StatefulSet with a headless Service. The gateway must choose workers itself (health, warm models, stickiness, canary). A normal Service would load-balance at L4 underneath it and defeat all of that — the headless Service exposes each pod by stable DNS name instead.
• The gateway discovers workers by DNS. It resolves the headless Service on its health-check cadence, so pods added or removed by an autoscaler join and leave the pool automatically — no gateway restart, no config edit. An empty DNS answer holds the previous pool (a resolver blip must not become an outage).
• The pool config is a Secret, not a ConfigMap — it holds the worker bearer tokens.
• A NetworkPolicy allows only the gateway to reach workers. That is also what makes it safe for workers to trust X-Forwarded-For from the gateway (AISUITE_TRUST_PROXY=1).
• The Ingress must be streaming-safe. Token streaming is Server-Sent Events: proxy buffering must be off and read timeouts generous (60 s+), or answers get cut mid-stream. The shipped Ingress annotations do this for nginx.

The zero-downtime contract

livenessProbe:
  httpGet: { path: /livez, port: 8080 }
  periodSeconds: 10
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
terminationGracePeriodSeconds: 40   # must be >= drain hold + shutdown budget

On SIGTERM the daemon flips /readyz to 503, holds for the drain window (AISUITE_DRAIN_SECONDS, default 8) so load balancers deregister it, then finishes in-flight requests within the shutdown budget (AISUITE_SHUTDOWN_SECONDS, default 30). Keep terminationGracePeriodSeconds at or above the sum, or the kubelet SIGKILLs a pod mid-drain — that single misconfiguration is the most common cause of dropped requests during rollouts.

Autoscaling

• Autoscaling needs three things together: an HPA, DNS worker discovery (so new pods actually receive traffic), and removing the static spec.replicas from the worker set (the HPA owns it — a GitOps sync that re-applies a replica count resets your fleet mid-peak, every sync, forever).
• CPU is the wrong signal for GPU inference — the GPU saturates while the CPU idles, so a CPU-based HPA packs more pods onto the bottleneck. Scale on the gateway's pool in-flight gauge (exposed via Prometheus metrics; use prometheus-adapter to feed it to the HPA).
• Without a metrics pipeline the HPA reports <unknown> and never fires — check kubectl describe hpa before assuming autoscaling works.

GPU nodes

• The GPU overlay/values add the device-plugin resource request (nvidia.com/gpu) and node selectors to the workers only — the gateway needs no GPU.
• Model stores live on each worker's PersistentVolumeClaim, so a rescheduled pod keeps its multi-gigabyte model cache instead of re-pulling it.
• GPU + autoscale are not stacked by default — combine them deliberately, with the in-flight-based HPA signal above.

Operational invariants worth pinning

• Mount worker API keys as a subPath file — mounting a Secret as a whole directory shadows the rest of the daemon's credentials directory and makes it read-only.
• Give the gateway a PVC too (or accept that a rescheduled gateway pod re-activates as a new install). The daemon releases its license seat on SIGTERM either way, so ephemeral pods don't strand seats.
• Rebuild and re-pull images after a license signing-key rotation — an image built before a new key was compiled in will refuse leases signed with it. That is the rotation contract working, not a bug.
• See Gateway & operations for monitoring the farm and Licensing & activation for the Secret-based activation details.