Batch Jobs

Token360 supports asynchronous batch inference for chat completions using the same workflow as the OpenAI Batch API. You upload a JSONL input file, create a batch job, poll until it completes, then download the output JSONL.

MVP scope:

Endpoint: POST /v1/chat/completions only (each JSONL line targets this URL).
Models: Use Token360 public model names in body.model (the same names as synchronous chat). Call GET /v1/batches/models to list batch-capable models for your account. Token360 routes each batch to one eligible SKU/provider at creation time.

Workflow

text

11. GET  /v1/batches/models          → discover supported model names
22. POST /v1/files (purpose=batch)   → upload input.jsonl
33. POST /v1/batches               → create job (input_file_id)
44. GET  /v1/batches/{batch_id}    → poll status until terminal
55. GET  /v1/files/{file_id}/content → download output or error JSONL

You may also configure an account-level batch webhook (see Webhook) or pass callback_url on create to override it for a single job.

Input JSONL format

Each non-empty line is one request object:


`custom_id`	Yes	Your stable ID to match output lines (order may differ).
`method`	Yes	Must be `POST`.
`url`	Yes	Must be `/v1/chat/completions`.
`body`	Yes	Chat completion body (OpenAI shape). Must include `model` and `messages`.

Example (two requests, same body.model on every line):

jsonl

1{"custom_id":"daily-greeting-001","method":"POST","url":"/v1/chat/completions","body":{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Say hi in one word."}],"max_completion_tokens":16}}
2{"custom_id":"daily-farewell-001","method":"POST","url":"/v1/chat/completions","body":{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Say bye in one word."}],"max_completion_tokens":16}}

Platform rules


Single model per file	All lines must use the same `body.model` string. Mixed models return `batch_multiple_models`.
No streaming	`stream: true` is rejected.
Line limit	Up to 50,000 requests per file.
File size	Input file max 200 MB; extension `.jsonl`, `purpose=batch`.
Completion window	Only `24h` is supported (OpenAI-compatible field).
Routing	Token360 picks one SKU/provider for the whole batch at create time and rewrites every line to the upstream model ID.

To run multiple models, create separate batch jobs (separate input files).

Discover supported models

Shell

1curl https://api.token360.ai/v1/batches/models \
2  -H "Authorization: Bearer sk-your-api-key"

Response (abbreviated):

JSON

1{
2  "object": "list",
3  "data": [
4    {
5      "id": "gpt-4o-mini",
6      "object": "model",
7      "supports_batch": true,
8      "display_name": "GPT-4o mini"
9    }
10  ]
11}

id is the value to use in JSONL body.model.

Create a batch (OpenAI SDK)

Python

1from openai import OpenAI
2
3client = OpenAI(api_key="sk-your-api-key", base_url="https://api.token360.ai/v1")
4
5# 1. Upload input
6batch_file = client.files.create(
7    file=open("requests.jsonl", "rb"),
8    purpose="batch",
9)
10
11# 2. Create batch
12batch = client.batches.create(
13    input_file_id=batch_file.id,
14    endpoint="/v1/chat/completions",
15    completion_window="24h",
16    metadata={"job": "nightly-summary"},
17    # optional: callback_url="https://your-app.example.com/webhooks/batch",
18)
19
20print(batch.id, batch.status)

Poll with client.batches.retrieve(batch.id) until status is completed, failed, expired, or cancelled. Then download output_file_id / error_file_id via GET /v1/files/{id}/content.

Batch statuses


`validating`	JSONL validated; job queued for upstream submit.
`in_progress`	Running on upstream provider.
`finalizing`	Downloading and processing output files.
`completed`	Success; `output_file_id` set when lines succeeded.
`failed`	Job failed validation or upstream error.
`expired`	Exceeded the 24h completion window.
`cancelling` / `cancelled`	Cancel requested or finished.

Poll GET /v1/batches/{batch_id} every 30–60 seconds for long jobs, or use webhooks.

Output JSONL

Completed jobs produce an output file in OpenAI batch result shape. Each line includes your custom_id and a response object. Token360 rewrites response.body.model back to your client model name when present.

Use custom_id to join results to your input — do not rely on line order.

Webhooks

When a batch reaches a terminal state (completed, failed, cancelled, expired), Token360 may POST to:

callback_url on the batch create request, if set, else
Your account default batch webhook URL (configure in the console under Batch jobs → Webhook settings).

Headers when a signing secret is configured:

Token360-Timestamp — Unix seconds
Token360-Signature — sha256= HMAC-SHA256 over {timestamp}.{raw_body}

Payload includes the batch object fields plus event (e.g. batch.completed). See Webhook for retry behavior.

Billing

Usage is metered from output JSONL response.body.usage per successful line, using the same SKU pinned at batch creation. Failed lines are not charged for completion tokens.