DEVELOPER DOCUMENTATION

Deferred batches

Queue work for a target discount. Poll for results without building a webhook receiver.

The worker requires enabled inference, a configured batch encryption key and a provider that enforces hard input/output price caps. Current live adapters do not provide that guarantee. Such requests remain waiting and expire if no qualifying route appears. A historical or observed floor cannot authorize a spend.

Submit an idempotent batch

Use the Batches workspace to submit JSON, inspect progress, cancel waiting work and download results, or call the API directly.

Send POST /v1/batches with bearer authentication and a unique Idempotency-Key header (8–128 visible ASCII characters). Keep that key unchanged when retrying the same submission; conflicting reuse returns 409.

{
  "min_discount": 90,
  "completion_window": "24h",
  "requests": [
    {
      "custom_id": "release-summary",
      "body": {
        "model": "your-model-id",
        "messages": [
          {
            "role": "user",
            "content": "Summarize this public release note."
          }
        ],
        "max_tokens": 64,
        "privacy": {
          "mode": "redact"
        }
      }
    }
  ]
}

The response supplies an ID, deadline, per-request statuses and results URL. The batch-level minimum discount is required. A per-request min_discount can increase it but cannot weaken it. Both token rates must meet the limit against fresh direct prices at dispatch, including every provider fallback.

Inspect, download and cancel

GET  /v1/batches/BATCH_ID
GET  /v1/batches/BATCH_ID/results
POST /v1/batches/BATCH_ID/cancel

Use the same owner’s API key for each call. Poll status every 15–30 seconds; download results as newline-delimited JSON. Each line contains a custom ID, status and available result with its HTTP status and reservation ID. Save completed results within 24 hours. Cancellation stops waiting items; already claimed work may finish.

Execution rules

The worker checks the market about every 15 seconds while idle and executes up to ten items per pass. It reserves credits only when a qualifying route exists and rechecks the deadline and submitting key before dispatch. Revoking that key cancels undispatched work. A deadline can be 1, 6 or 24 hours; the default is 24 hours.

Requests expire when the target never becomes available. Dispatched failures are not automatically retried. Interrupted work becomes uncertain; inspect Activity before resubmitting to avoid duplicate spending. An idempotency key deduplicates submissions while batch metadata is retained; it is not an eternal execution guarantee.

Limits: 1–50 inline requests, up to 1 MB per submission (subject to the gateway body limit), 200 pending items per owner, 10 submissions per minute, paid non-streaming text only, and at most 4,096 output tokens per request (default 1,024). Privacy runs before input persistence. Invalid or privacy-rejected submissions queue nothing. Funds are not held while waiting, so a request can fail if credits are unavailable when its target arrives.

The queue executes at the first eligible check. It cannot know the lowest future price or guarantee a 90–95% saving on every model. Image generation, tools, native provider batch jobs and webhooks are not part of this endpoint.