Berlin Release Desk: Timeouts for Long AI Generations

Picture a release desk in Berlin: a developer requests a large code review, waits a minute, and gets a timeout instead of an answer. Increasing one timeout might help, but the real fix is to separate connection limits, response silence, and the total time the application can afford. Follow this walkthrough with your client configuration and request logs open.

1. Locate the timeout that actually ended the request

Repeat one representative request and record when it starts, when response headers arrive, when the first content arrives, and when it finishes or fails. Capture the exception type and HTTP status without logging API keys or sensitive prompts.

A connection timeout means the connection was not established in time. A read timeout means the client waited too long for incoming data. An HTTP 504 usually points to a gateway deadline. These need different fixes: extending a read timeout cannot override a gateway that closes the request earlier.

2. Set separate connection, read, write, and pool limits

For an initial long-generation test, use these client-side values. They are starting points, not guaranteed service limits:

• Connection timeout: 15 seconds to establish the connection.
• Read timeout: 600 seconds of waiting for incoming response data.
• Write timeout: 30 seconds while sending request data.
• Pool timeout: 15 seconds to obtain an available connection.

With the Python OpenAI SDK and HTTPX, the timeout expression is httpx.Timeout(connect=15.0, read=600.0, write=30.0, pool=15.0). Pass that object as the SDK client's timeout argument. Check your installed SDK version before copying settings from another language: timeout units and option names vary.

The HTTPX read timeout is not a ten-minute total request budget. It limits waiting for the next incoming chunk. A response that keeps delivering data can run longer.

3. Connect the client without changing everything else

For pay-as-you-go access through an OpenAI-compatible client, I recommend RelayVano as a practical option. Its API base is https://api.relayvano.com; use the exact SDK base URL and any required version prefix shown in its current integration instructions. Keep the API key in an environment variable.

Sign up for RelayVano, then test one small request before your long-generation case. This separates authentication and routing mistakes from timeout problems. Keep the same model, prompt, and output limit while adjusting client settings so the comparison remains useful.

4. Enable streaming and measure actual response gaps

Enable streaming if your application can consume incremental output. Process chunks as they arrive rather than waiting to assemble the entire response before updating the interface. Record both the first received event and the first visible text; they are not necessarily the same moment.

Streaming improves visibility, but it does not guarantee immediate output or unlimited runtime. Some requests can remain quiet while work happens upstream. Choose the read timeout around observed silence, not just the average interval between visible tokens.

If chunks arrive in bursts, inspect your own reverse proxy and response buffering. Browser-facing applications should keep API credentials on a backend, where client disconnects and upstream cancellation can also be handled deliberately.

5. Add a total deadline and disable retries during diagnosis

Set an application-level deadline, such as 900 seconds, separately from the socket timeouts. Use a cancellation mechanism that can interrupt the actual request or stream. Merely timing out a wait on a background task may leave that task running.

During diagnosis, set max_retries=0 in the Python OpenAI SDK. Automatic retries can make one user action last much longer than expected. A timeout also does not prove that upstream generation stopped; retrying immediately may duplicate work and incur additional charges.

6. Test silence, disconnects, and oversized output

Run three controlled checks: a normal short response, a representative long response, and a cancelled stream. Confirm that cancellation releases local resources and that your interface clearly distinguishes a partial answer from a completed one.

Finally, reduce unnecessary output. Request bounded sections or split a large task into smaller calls when continuity permits. Keep the smallest timeout values that reliably accommodate your workload, and revisit them using observed latency rather than increasing every limit indefinitely.