Skip to main content
Inference runs through the OpenAI-compatible chat completions endpoint on the data plane. Authenticate every request with an org API key in the Authorization header:
Endpoint
The request body follows the OpenAI chat completions schema: a model ID from the catalog, a messages array, and optional parameters. Any OpenAI SDK works unchanged, see the quickstart for client setup in Python and TypeScript. Two companion endpoints ride alongside: GET /models (the live catalog) and the legacy POST /completions (raw-prompt completions, streaming included).

The response envelope

A non-streamed response is a single JSON object:
  • choices[0].message, the assistant reply. Reasoning models add a reasoning field alongside content; tool use adds tool_calls.
  • finish_reason, stop (natural end or stop sequence), length (hit max_tokens, the reply is truncated), or tool_calls (the model wants your code to run a tool).
  • usage, the token counts your organization is billed on. Depending on the serving worker, it may also carry prompt_tokens_details.cached_tokens (see prompt caching) and completion_tokens_details.reasoning_tokens.

Parameters

Alongside the standard OpenAI parameters (temperature, top_p, frequency_penalty, presence_penalty, max_tokens, seed, up to 4 stop sequences), models accept top_k and min_p extensions, pass those via extra_body in the OpenAI SDKs. Per-model support is listed in supported_sampling_parameters on GET /models.

Rate limits

New self-serve organizations start on Basic at 10 requests per minute and 3 concurrent requests per model. Once a credit purchase of at least $5 settles, they automatically get 600 requests per minute and 50 concurrent requests per model, while staying on Basic. You do not need to save a credit card. Custom invitations and custom rate limits keep their settings. Input tokens, output tokens, and concurrent requests have separate limits; see Plans & rate limits for the full table and payment requirements.

Errors

Error responses are JSON and carry a request_id, log it, and include it when you contact support; it is how a specific request gets found on the network.
A 402 is an account state, not a transient fault: requests keep failing until the balance is topped up or the unpaid invoice is paid. If you run production traffic on a prepaid balance, enable auto-recharge so a busy night doesn’t become an outage.

Retry guidance

  • Retry 429 and 5xx with exponential backoff and jitter; cap the attempts and surface the request_id when giving up.
  • Do not blind-retry 400/401/402/404, they fail the same way until the request, key, or balance changes.
  • Once a response starts streaming, a failure ends the stream instead of returning an error object, see Streaming for how to detect an incomplete stream.