Retry Rules for a Support Bot’s Dropped Stream

Your team is adding streamed answers to a customer support bot before Friday’s release. During testing, an answer starts normally: “To reset your account, open…” Then the text stops. The loading indicator keeps spinning, and the customer never gets the remaining steps.

Streaming sends an answer in small pieces while the model generates it. That makes the interface feel responsive, but it creates a tricky failure case: some output has already arrived when the connection breaks. A retry needs to account for that partial result.

A dropped connection leaves the request’s outcome uncertain

A stream can stop because of a network interruption, a proxy timeout, an upstream error, or an application closing the connection. Your client sees the interruption, but it may not know whether generation also stopped on the server.

This matters because sending the same request again usually starts a new generation. The second answer may use different wording, and the first attempt may already have consumed billable usage. Treat a retry as another attempt unless the API explicitly documents a way to resume the original request.

Also separate a connection ending from an answer finishing. Use the completion signal documented for your endpoint or exposed by your SDK. A closed connection alone does not prove success, and a normal completion can still report that output stopped at a token limit.

The first visible text changes the retry decision

For a first implementation, divide failures by what your application has already received and displayed:

• Before any answer text arrives, retry a transient failure within a small attempt and time budget.
• After answer text appears, preserve it, mark the response as interrupted, and offer a clearly labeled retry.
• After a confirmed successful completion, keep the answer. Do not regenerate it just because later interface cleanup fails.

This keeps an automatic retry from quietly replacing words the customer was already reading. If your product automatically restarts partial answers, visibly separate the attempts and tell the reader what happened.

A useful interrupted state says: “The connection ended before this answer finished. Try again.” Keep the partial answer available, especially when it contains information the customer may want to copy.

Retry transient failures with a bounded delay

Temporary connection failures, rate limits, and some server errors can justify another attempt. Invalid credentials, malformed requests, and unsupported parameters generally require a fix before retrying. Check the endpoint’s documented error behavior instead of retrying every failure.

A reasonable starting policy allows three attempts total, with roughly one second before the second attempt and two seconds before the third. Add a small random variation, called jitter, so many clients do not reconnect at exactly the same moment. Honor a documented Retry-After value when present, provided it fits your overall deadline.

Set separate limits for waiting for the stream to start, waiting between chunks, and the entire operation. A gap between chunks does not automatically mean failure; some requests need longer pauses. Choose limits from observed behavior, and cancel pending retries when the customer leaves or presses Stop.

Partial answers and tool actions need separate handling

Do not append a fresh generation directly to an interrupted answer. Repeating the original prompt can produce a different opening, duplicate paragraphs, or conflicting instructions. Start a separate answer instead.

You can ask a model to continue from the saved text, but that is a new request with context, not a guaranteed continuation of the original stream.

Be stricter when the assistant can issue refunds, create tickets, or change records. Never execute incomplete tool arguments. Record completed actions and use the action service’s documented idempotency mechanism where available. A retried model request must not silently cause a second refund.

A release check for the support bot

Before shipping, deliberately disconnect a stream before its first text, halfway through an answer, and after a completed tool action. Confirm that retries stop at their limit, partial text stays readable, and actions happen only once. Log attempt identifiers, completion status, and available usage data to investigate failures and costs.

For developers connecting this flow to AI models, RelayVano is worth considering as a pay-as-you-go relay with an OpenAI-compatible API base at https://api.relayvano.com. Sign up for RelayVano, then test your application’s interruption handling through the endpoint before releasing the support bot.