Chat completions API — streaming, tools, JSON output and vision

POST /v1/chat/completions takes the same request as OpenAI's Chat Completions API and returns the same response shape, streamed or not.

curl http://<server>/v1/chat/completions \
  -H "Authorization: Bearer $AISERVER_KEY" -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.2:3b",
    "messages": [
      {"role": "system", "content": "You answer in one sentence."},
      {"role": "user", "content": "What is a GPU?"}
    ],
    "temperature": 0.3
  }'

Request parameters

Response

{
  "id": "chatcmpl-…", "object": "chat.completion", "created": 1759650000, "model": "enginea/llama3.2/3b",
  "choices": [ { "index": 0, "message": { "role": "assistant", "content": "A GPU is …" }, "finish_reason": "stop" } ],
  "usage": { "prompt_tokens": 24, "completion_tokens": 18, "total_tokens": 42 }
}

finish_reason is stop, length or tool_calls.

Streaming

With "stream": true the answer arrives as chat.completion.chunk events with a delta, ending with data: [DONE]. Every OpenAI SDK handles this unchanged.

stream = client.chat.completions.create(model="llama3.2:3b", stream=True,
    stream_options={"include_usage": True},
    messages=[{"role": "user", "content": "Write a haiku about servers."}])
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

If you put a proxy in front of AI Server, turn off response buffering or the stream arrives all at once. When an operator has turned on content rules for answers, the server checks the complete answer before sending it, so the stream arrives in one burst at the end.

Tool calling

Send tools; the model answers with tool_calls instead of text when it wants one. Your code runs the tool and sends the result back as a tool message; the server keeps the whole conversation, in order, and the model continues.

tools = [{"type": "function", "function": {
    "name": "get_weather", "description": "Current weather for a city",
    "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}]
messages = [{"role": "user", "content": "Do I need an umbrella in Hong Kong?"}]

reply = client.chat.completions.create(model="qwen2.5:7b", messages=messages, tools=tools)
call = reply.choices[0].message.tool_calls[0]
messages.append(reply.choices[0].message)
messages.append({"role": "tool", "tool_call_id": call.id, "content": '{"rain_mm": 12}'})
final = client.chat.completions.create(model="qwen2.5:7b", messages=messages, tools=tools)
print(final.choices[0].message.content)

• Use a general instruct model of 7–8B parameters or more for multi-step agents. Coding-tuned and very small models tend to write the call as plain text.
• tool_choice: "required" forces a structured call; naming a function forces that one.
• Streaming returns tool calls as tool_calls deltas.

JSON output

{"type": "json_object"} asks for valid JSON; {"type": "json_schema", "json_schema": {"name": …, "schema": {…}}} passes your schema to engines that can constrain their output to it. Always parse and validate the answer — smaller models can still produce JSON that does not match the schema.

Images in a message

Send images as image_url parts with a base64 data URL. The model must support images (vision); others answer 400.

{"role": "user", "content": [
  {"type": "text", "text": "What is in this photo?"},
  {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,/9j/4AAQ…"}}
]}

Remote https:// image URLs are passed on only to cloud providers; local models need the image inline.

Limits and errors

• Over a rate limit, quota or the Free allowance: 429 with Retry-After.
• A model the key may not use: 403. A model that is not installed: 404 model_not_found.
• A deprecated model answers with the X-AI-Model-Deprecated header.
• Behind a gateway, a request that fails on one worker before any output is sent is retried on another.

See the error reference and rate limits.

Questions