DEVELOPER DOCUMENTATION

Latency & benchmarks

Measure the wait, from gateway preparation to the first token and completion.

Every request has a receipt

Activity shows timings for newly recorded requests and p50/p95 summaries for successful requests in the loaded page. Filter by model or provider. Old receipts without timing fields stay available; they do not count as measured samples.

Gateway preparation: handler entry to provider dispatch, including authentication, privacy checks, routing and credit reservation.
Time to first token: handler entry to the first nonempty streamed content or refusal delta. Role, heartbeat and usage events do not count. Non-streaming requests have no TTFT measurement.
Upstream: dispatch to completion, error or cancellation, including provider network time and stream backpressure.
Total: handler entry to completion observation. Excludes earlier request-body middleware, final ledger settlement and the client’s network travel.

The Server-Timing header exposes gateway preparation and, for completed non-streaming responses, upstream time. Streaming headers cannot contain future timings; retrieve the settled receipt through GET /v1/receipts/:reservationId or Activity. Failed and cancelled requests retain their outcomes.

Compare routed and direct requests

The gateway includes a bounded benchmark CLI. Set GPU_ROUTER_API_KEY and DIRECT_API_KEY in your shell environment. Use base URLs ending in /v1 and the exact equivalent model on both endpoints.

pnpm --filter @gpu-protocol/compute-gateway benchmark
# Plan only; sends no requests.

pnpm --filter @gpu-protocol/compute-gateway benchmark --run \
  --gateway https://gpu-router-accounts-production.up.railway.app/v1 \
  --direct https://YOUR_DIRECT_PROVIDER/v1 \
  --model your-model-id --direct-model EXACT_DIRECT_MODEL_ID \
  --samples 5 --max-tokens 64 --output latency-report.json

This example makes 12 sequential calls, including one excluded warmup pair. It alternates endpoint order and uses the same prompt within each pair. The JSON report includes client-observed TTFT, total time, p50/p95, success counts, token usage and paired differences. It retains no keys, prompt text or generated output.

Routed-versus-direct differences also include provider, network, caching and load variation. They do not isolate router overhead. No live benchmark result or “negligible overhead” guarantee is published here. Small samples, especially p95, are exploratory.