Session Affinity in the Token360 API for Multi-Turn Chat

By Yuki Yu

An editor is six turns into revising a product description. Your app sends the same public model name each time, but the conversation also depends on state produced by an upstream provider. How can the next turn prefer the provider that handled the last one?

Give the conversation a stable, opaque session ID. Token360 can use that value to prefer the same eligible upstream provider and credential on later requests. Your application still owns the conversation history. The engineering decision is what the user should experience when the preferred provider becomes unavailable. Token360 routing and reliability.

Read first. If you are still choosing which public model to call, start with the model routing guide. The examples below keep the public model fixed across turns.

Where should a chat application create the session ID?

Create it when the user starts the conversation and store it with the conversation record. Read the stored value when building each subsequent request, including requests handled by a different application worker. A random identifier generated inside a per-request helper changes on every turn.

For example, suppose an editor opens one chat for catalog copy and another for translations. Assign each chat its own value. If the editor reloads the catalog tab, restore that conversation's value. A stable account-wide identifier would make the two chats appear identical to an integration check that groups outbound requests by session ID. That is an observability and routing-design problem. It does not, by itself, demonstrate that conversation content passed between users.

Keep the session value opaque. Store the associations you need for internal debugging in your own records. Treat the request ID as a separate field so an operator can find one failed attempt without confusing it with every call in the conversation.

Send the same ID on successive calls

Choose a single owner for setting the ID. Token360 accepts the x-session-id header or the top-level session_id body field. The body value wins if both are supplied. The following two-turn request shows the header approach with an illustrative conversation and a server-side key variable. Session identifier reference.

curl https://api.token360.ai/v1/chat/completions \
  -H "Authorization: Bearer $TOKEN360_API_KEY" \
  -H "Content-Type: application/json" \
  -H "x-session-id: chat-7d9b1f" \
  -d '{
    "model": "gpt-6.1-sol",
    "messages": [
      {"role":"user","content":"Suggest a short title for a blue desk lamp."},
      {"role":"assistant","content":"Blue Hour Desk Lamp"},
      {"role":"user","content":"Keep Blue Hour and shorten the rest."}
    ]
  }'

The sample supplies the prior messages explicitly. In your application, insert the actual prior response and any tool or protocol fields required to continue that conversation. Resolve the model ID against your account before running the example. For agent workloads, test a complete second turn, including the return from a tool where applicable. Token360 model capability checks.

When should you allow a different provider?

For an ordinary text exchange that your application can reconstruct from its own records, continuity of service may take priority. When the next turn depends on provider-specific state, a controlled interruption may be easier to recover from than an attempt to replay that state elsewhere.

Situation	Documented behavior	Application decision
First call, no remembered route	Token360 chooses among eligible candidates. A success can establish a preference.	Keep the session ID for turn two.
Remembered provider unavailable, fallback enabled	Another eligible route may serve the call. A successful switch updates the preference.	Check that the next turn can resume with the available state.
Remembered provider unavailable, allow_fallbacks: false	The call returns HTTP 503 under hard affinity.	Offer a bounded retry or an explicit restart path.

Hard affinity takes effect once there is a remembered preference. Successful requests refresh a ten-minute inactivity window. After it expires, plan for a fresh routing decision. Affinity is scoped by account, public model and session ID. Token360 affinity lifecycle.

Consider a chat paused while its user checks a reference. The user reasonably expects their transcript to survive lunch. Store that transcript according to your product's retention policy and rebuild the next request from the saved state. Treat route continuity and application history as two separate requirements when designing the resume button.

What can an offline check reveal before you spend on inference?

Inspect the identifier that will actually leave the final request builder. We created two synthetic conversations with three turns each, held the account and public model constant, and ran four client configurations through a local Python audit. The script records the effective ID after header/body precedence, counts changes between adjacent turns and finds IDs used by both conversations.

Configuration	Distinct IDs	Within-chat changes	IDs shared across chats
One ID per conversation	2	0	0
Fresh ID per request	6	4	0
One application-wide ID	1	0	1
Correct headers plus a stale body default	1	0	1

The final row is the easy mistake to miss. A unit test for the header helper would see a different ID for each chat and pass. The request builder also attaches a body default, so both conversations ultimately present the same effective value. Capture a sanitized representation after middleware has finished assembling the request. Compare the expected conversation association there.

Example image fixtures: . Run python3 session_audit.py to rebuild the four traces. The fixture exercises client-side ID handling only. Live route selection, provider state and cache usage require separate observations.

Copy session_audit.py:

"""Inspect synthetic outbound session identifiers for one account and public model."""
import csv
from collections import defaultdict
from pathlib import Path

def make_trace(pattern):
    for conversation in ('A', 'B'):
        for turn in range(1, 4):
            header = f'chat-{conversation}'
            body = ''
            if pattern == 'per_turn':
                header += f'-{turn}'
            if pattern == 'shared':
                header = 'shared-value'
            if pattern == 'body_override':
                body = 'old-default'
            yield conversation, turn, header, body

def audit(trace):
    sequence = defaultdict(list)
    owners = defaultdict(set)
    for conversation, turn, header, body in trace:
        effective = body or header  # Fixtures contain only valid, nonempty IDs when supplied.
        sequence[conversation].append(effective)
        owners[effective].add(conversation)
    changes = sum(a != b for ids in sequence.values() for a,b in zip(ids,ids[1:]))
    return len(owners), changes, sum(len(group)>1 for group in owners.values())

if __name__ == '__main__':
    directory = Path(__file__).parent
    with (directory/'audit.csv').open('w',newline='') as dest:
        writer=csv.writer(dest)
        writer.writerow(['scenario','distinct_effective_ids','within_chat_changes','ids_shared_between_chats'])
        for pattern in ('per_chat','per_turn','shared','body_override'):
            result = audit(make_trace(pattern))
            writer.writerow([pattern,*result])
            print(pattern,*result)

Copy audit.csv:

scenario,distinct_effective_ids,within_chat_changes,ids_shared_between_chats
per_chat,2,0,0
per_turn,6,4,0
shared,1,0,1
body_override,1,0,1

How can you tell whether affinity helped in production?

Start with outcomes your application can observe. Check whether a second turn completes with the required context, how often users encounter a recoverable interruption and how long the complete turn takes. Keep a separate field for observed cache usage when the response exposes it. A repeated model name or a fast reply alone cannot identify the serving route.

For a small pilot, choose one multi-turn task and hold the model, prompt prefix and turn sequence steady. Record the session and request identifiers separately, alongside response status, timing and available usage fields. Compare similar turn positions and input lengths. A first turn with a short prompt and a tenth turn with a long transcript are poor candidates for a direct latency comparison. Measuring API latency.

Provider affinity can give compatible repeated work an opportunity to use a prompt cache. Cache eligibility and savings depend on the serving system and the request. OpenRouter describes provider-specific caching considerations in its prompt caching guide. For a Token360 pilot, compare actual recorded charges before estimating savings. Token360 billing and usage.

Read next. Decide how your application should handle an interrupted turn or retry after tools have run. The API fallback guide covers those recovery decisions. Related product documentation is available in Token360 resources.

What does a session identifier not preserve?

A session identifier is a routing input, not a transcript, a user credential or an assurance that every later call will hit a cache. Keep the conversation history and authorization in your application. Send the context required by the selected protocol even when the session identifier stays unchanged.

When a conversation must move to another route, decide whether its saved context can be replayed there. Ordinary messages and provider-specific continuation state may have different portability. Test recovery with the actual fields your application stores; an offline check that two requests carry the same identifier cannot establish upstream continuity.

For support, retain the conversation reference, request times, public model name and observed outcome. Do not place an API key or private conversation text inside the session identifier. The Token360 routing documentation remains the reference for the supported affinity controls.

A two-turn record needs route evidence as well as a session ID

Use one opaque session identifier for both turns and preserve the conversation history separately. Turn one can ask the model to remember a fictional project label; turn two can ask it to repeat that label while including the required prior messages. Correct recall demonstrates the supplied conversation path, not that affinity or a prompt cache worked.

Record	What it answers
Same account, public model and session value	Was the intended affinity scope reused?
Sanitized messages for each turn	Was the history supplied correctly?
Request identifiers and available route metadata	Can the serving path be investigated?
Returned cache usage, if exposed	Is there evidence for a cache hit rather than an assumption?

The routing documentation describes affinity as a preference constrained by routing policy and availability. If the account exposes no route evidence, report “route continuity unverified” even when both answers are correct. A local validator can inspect the sent session value; it cannot observe which provider served those requests.

Frequently asked questions

Should a new turn get a new session ID?

Reuse the conversation's existing value for later turns in that conversation. Assign a distinct value to a new conversation or independent agent run.

What happens if the header and body disagree?

Token360 uses the body field. Give one integration layer responsibility for writing the ID and inspect the assembled request when debugging.

Can a session ID restore the chat transcript?

The ID communicates a routing preference. The application must supply or restore the context required by the selected conversation protocol.

Will disabling fallback select a specific provider on the first call?

When no preference exists, ordinary eligible-route selection applies. Hard affinity constrains subsequent routing after a preference has been established.

Does a successful local audit establish that a cache was hit?

The audit establishes how your client handles identifiers. Inspect live usage evidence and settled charges to evaluate caching.

Can we reuse the same ID after changing models?

You can retain an application conversation identifier, but the public model is part of the affinity scope. Treat the changed model as a separate routing context.

Editorial revision: November 4, 2026. Test dates and dated source snapshots remain as stated in the article.