Batch Jobs

Token360 supports asynchronous batch inference for chat completions using the same workflow as the OpenAI Batch API. You upload a JSONL input file, create a batch job, poll until it completes, then download the output JSONL.

MVP scope:

• Endpoint: POST /v1/chat/completions only (each JSONL line targets this URL).
• Models: Use Token360 public model names in body.model (the same names as synchronous chat). Call GET /v1/batches/models to list LLMs you can submit to POST /v1/batches (at least platform fan-out). supports_provider_batch is a separate flag for upstream native Infer.
• Execution paths: Batches run via provider native Batch Infer when available, or via platform fan-out (Token360 executes each line through chat completion). Create-time execution selects the path (default auto).

Console

In the Token360 console, open Batch jobs (/me/batches):

API-only workflows below work the same way; the console uses the same /v1/files + /v1/batches APIs.

Dual execution

Response fields (Token360 extensions):

Console and API both surface execution on the batch object so you can tell which path ran.

Workflow

11. GET  /v1/batches/models          → LLMs you can POST /v1/batches (see supports_provider_batch)
22. POST /v1/files (purpose=batch)   → upload input.jsonl
33. POST /v1/batches               → create job (input_file_id, optional execution)
44. GET  /v1/batches/{batch_id}    → poll status until terminal
55. GET  /v1/files/{file_id}/content → download output or error JSONL

Webhook / callback: pass callback_url on create, or set the account default under console Batch jobs → Webhook. Per-job URL wins. If neither is set, poll only. Delivery, signing, and payload (including execution / usage_billing) are documented under Webhook.

Input JSONL format

Each non-empty line is one request object:

Example (two requests, same body.model on every line):

1{"custom_id":"daily-greeting-001","method":"POST","url":"/v1/chat/completions","body":{"model":"claude-opus-5","messages":[{"role":"user","content":"Say hi in one word."}],"max_completion_tokens":16}}
2{"custom_id":"daily-farewell-001","method":"POST","url":"/v1/chat/completions","body":{"model":"claude-opus-5","messages":[{"role":"user","content":"Say bye in one word."}],"max_completion_tokens":16}}

Platform rules

To run multiple models, create separate batch jobs (separate input files).

Discover supported models

1curl https://api.token360.ai/v1/batches/models \
2  -H "Authorization: Bearer sk-your-api-key"

Response (abbreviated):

1{
2  "object": "list",
3  "data": [
4    {
5      "id": "claude-opus-5",
6      "object": "model",
7      "supports_batch": true,
8      "supports_provider_batch": false,
9      "display_name": "Claude Opus 5"
10    }
11  ]
12}

id is the value to use in JSONL body.model.
supports_batch means the model can run as a Token360 batch (provider and/or platform).
supports_provider_batch is true when an upstream native Batch Infer route is available.

Create a batch (OpenAI SDK)

1from openai import OpenAI
2
3client = OpenAI(api_key="sk-your-api-key", base_url="https://api.token360.ai/v1")
4
5# 1. Upload input
6batch_file = client.files.create(
7    file=open("requests.jsonl", "rb"),
8    purpose="batch",
9)
10
11# 2. Create batch (execution defaults to auto)
12batch = client.batches.create(
13    input_file_id=batch_file.id,
14    endpoint="/v1/chat/completions",
15    completion_window="24h",
16    metadata={"job": "nightly-summary"},
17    extra_body={
18        "execution": "platform",
19        "callback_url": "https://your-app.example.com/webhooks/batch",
20    },
21)
22
23print(batch.id, batch.status)

Poll with client.batches.retrieve(batch.id) until status is completed, failed, expired, or cancelled. Then download output_file_id / error_file_id via GET /v1/files/{id}/content.

Batch statuses

expires_at / completion_window=24h is an estimate (OpenAI-compatible field). Large queues can run longer on the platform path.

Poll GET /v1/batches/{batch_id} every 30–60 seconds for long jobs, or use webhooks. Cancel non-terminal jobs with POST /v1/batches/{batch_id}/cancel.

Output JSONL

Completed jobs produce an output file in OpenAI batch result shape. Each line includes your custom_id and a response object. Token360 rewrites response.body.model back to your client model name when present.

Use custom_id to join results to your input — do not rely on line order.

Webhooks

When a batch reaches a terminal state (completed, failed, cancelled, expired), Token360 may POST to:

1. callback_url on the batch create request, if set, else
2. Your account default batch webhook URL (configure in the console under Batch jobs → Webhook).

Headers when a signing secret is configured:

• Token360-Timestamp — Unix seconds
• Token360-Signature — sha256= HMAC-SHA256 over {timestamp}.{raw_body}

Payload includes the batch object fields plus event (e.g. batch.completed). See Webhook for retry behavior.

Billing

• Platform execution (execution=platform, or auto resolved to platform): successful lines are billed at the same rates as synchronous chat, including the account’s enterprise / VIP price factor. There is no extra 50% batch discount on this path. Failed / cancelled lines are not charged.
• Provider execution (execution=provider, or auto resolved to provider): usage is metered from output JSONL; provider-native batches may apply a batch discount when configured on the account/SKU path. Check usage_billing on the batch object for list vs charged amounts.
• Reconcile a line with GET /v1/billing/{batch_id}-0000 (zero-padded ordinal). The batch object’s usage_billing is the sum of successful lines.

Related API reference

• List batch-supported models
• Upload batch input file
• Create batch
• Retrieve batch
• List batches
• Cancel batch
• Download batch file content