A Berlin API Team’s Rules for Broken AI Streams

At 4:45 p.m., your team’s chat interface stops halfway through an answer. The browser still shows a spinner. Your server has received text, but no completion signal. Retrying sounds obvious until the replacement answer repeats three paragraphs—or repeats a tool action.

Use this numbered walkthrough to turn that ambiguous failure into explicit retry rules. Keep your request handler, stream parser, and application logs open while you work.

1. Record the request before opening the stream

Create a logical operation ID for the user’s request and a separate attempt ID for each network attempt. Store the model, request settings, start time, and conversation version. Keep prompts and credentials out of routine logs; use an appropriately protected record if replay requires the original payload.

Track three milestones separately: response headers received, first content received, and terminal completion observed. An HTTP 200 means the connection started successfully, not that generation finished.

Also count the bytes or text fragments delivered to the user. That number determines whether a retry can remain invisible or needs a visible recovery state.

2. Identify completion from protocol events, not silence

Check the streaming protocol your endpoint actually returns. For server-sent events, network reads do not necessarily line up with event boundaries. Buffer incomplete frames and parse complete events before interpreting them.

Define success using the endpoint’s documented terminal signals. OpenAI-compatible interfaces can use different event shapes depending on the API. Do not treat a closed socket, an empty read, or a pause as proof of completion.

If the connection ends without the required terminal signal, mark the attempt incomplete even when the visible text ends with a plausible sentence.

3. Separate retryable failures from rejected requests

Build a small decision table beside your handler:

• Authentication, permission, and invalid-input errors: stop and fix the request.
• Rate limits and transient service errors: consider a bounded retry, respecting Retry-After when provided.
• Connection resets and read timeouts: consider a retry, but first check whether content or tool instructions already arrived.
• User cancellation: stop; do not silently restart work the user canceled.

Configure connection, first-event, idle, and overall deadlines separately. Heartbeats may show that a connection is alive without proving that useful generation is progressing.

4. Choose a recovery policy for visible partial text

Before any content reaches the user, a retry can usually replace the failed attempt transparently. Once text is visible, choose an explicit interface policy: discard and regenerate, preserve the partial answer with a retry button, or display a clearly labeled replacement answer.

Do not append a fresh generation directly to the old text. A repeated request is not a continuation cursor, and its wording can diverge immediately.

If you offer a separate “continue” operation, send the partial answer as context and explain that overlap or omissions remain possible. That is a new generation strategy, not guaranteed transport-level resume.

5. Protect tool actions before replaying anything

Buffer tool-call arguments until they are complete and validated. Never execute a tool from partial streamed arguments.

Record each authorized action and its result in durable application state. If a payment, message, or deployment may already have executed, reconcile its status before retrying. Use stable application-level action IDs and downstream idempotency support where available.

A model request ID alone does not make side effects safe. Do not assume an API accepts idempotency keys or resumes streams unless its documentation explicitly says so.

6. Bound retries and make their cost visible

Start with one initial attempt and at most two retries for eligible failures, within an overall deadline. Use exponential backoff with jitter—for example, randomized waits around one and two seconds—and honor longer server-directed delays when your deadline permits.

Check your SDK’s automatic retry settings so application retries do not multiply hidden attempts. A broken connection also does not prove generation stopped upstream; failed attempts may still incur charges.

For developers evaluating a pay-as-you-go relay, I recommend RelayVano for this workflow: sign up and test your OpenAI-compatible SDK against https://api.relayvano.com, using the documented endpoint path. Verify streaming behavior and retry settings rather than assuming compatibility supplies recovery automatically.

7. Inject a midstream failure before shipping

Use a local proxy or test harness to cut the connection after several events. Repeat before the first event, during a partial frame, and after a tool action. Confirm that the UI leaves its loading state, retries stay bounded, replacement text is identifiable, and executed actions are not duplicated.

Log the logical operation ID across every attempt. Your release check is simple: every broken stream ends in a completed answer, an explicit recoverable failure, or a cancellation—not an endless spinner.