A 429 Retry Plan for a Berlin Batch Deadline

1. Capture the 429 before restarting the batch

Picture a Berlin development team preparing an overnight batch of AI summaries. Four workers pass the smoke test. Forty workers trigger HTTP 429 responses. Restarting the batch immediately only sends another burst into the same limit.

A 429 means the service is rejecting requests because a limit has been reached. That might involve request frequency, token throughput, concurrent requests, or an account restriction. Exponential backoff helps with temporary limits; it does not create more quota.

At your desk, capture one failed response. Record the status, error type, request ID, timestamp, model, attempt number, and any rate-limit headers. Avoid logging API keys or sensitive prompts. Read the error body before deciding whether another attempt is useful.

2. Separate temporary throttling from exhausted quota

Create a retry decision before writing the delay logic. Treat a temporary rate-limit response as potentially retryable. If the error indicates exhausted credits, a hard spending limit, or an account restriction, stop automatic retries and surface an actionable message instead.

• Temporary throttling: retry within a fixed budget.
• Exhausted quota or billing restriction: pause the job and notify its owner.
• Unclear 429: allow bounded retries, then preserve the error for investigation.

Check your SDK configuration now. Some clients already retry requests. Adding five application retries around an SDK that also retries can multiply network attempts unexpectedly. Choose one layer to own the retry budget, or explicitly account for both.

3. Calculate exponential backoff with full jitter

Start with a one-second base delay and a 30-second exponential cap. For retry index n, beginning at zero, calculate window = min(30 seconds, 1 second × 2^n). Then choose a random delay uniformly between zero and that window.

Your first five retry windows are therefore 0–1, 0–2, 0–4, 0–8, and 0–16 seconds. This is full jitter: each worker chooses its own waiting time instead of waking at identical intervals.

If the server supplies Retry-After, parse it as either a number of seconds or an HTTP date. Treat a valid value as a minimum wait. Use delay = max(server wait, jitter delay), optionally adding a small random spread when many workers receive the same server wait. Do not truncate that minimum to your 30-second exponential cap.

4. Bound retries with attempts and a deadline

Set two limits: at most five retries after the original request, and a 90-second overall deadline for this example. Those are starting values for a batch workload, not universal defaults. An interactive application usually needs a shorter budget.

Before every sleep and request, check the remaining time using a monotonic clock. If the required wait leaves no useful request time, stop rather than retrying early. Set each request timeout to fit inside the remaining budget, and make sleeps cancellable so a stopped job does not keep retrying.

Keep the request payload stable across attempts. Do not assume that a timeout or broken connection has the same safety properties as a received 429. For operations with side effects, confirm the endpoint's idempotency behavior before replaying them.

5. Limit worker traffic before it reaches the API

Backoff handles rejection after it happens. A shared queue helps prevent it. Return the Berlin batch to four concurrent requests, measure throughput, and increase concurrency gradually. Keep delayed retries in the same admission-control system as new work.

If errors track token volume rather than request count, reduce requested output length where practical and avoid oversized batches. Coordinate limits across processes; a semaphore inside one worker cannot control twenty separate workers.

If your application needs pay-as-you-go access to AI models through an OpenAI-compatible SDK, RelayVano is worth considering, with the API base https://api.relayvano.com. Sign up to test it with a small workload, while retaining the same bounded retry policy rather than assuming a relay eliminates upstream limits.

6. Test the retry policy before the overnight run

Use a mock endpoint to return two temporary 429s followed by success. Then test Retry-After, exhausted quota, cancellation, and repeated failures. Verify that five retries means six total attempts and that the deadline always wins.

Track first-attempt 429 rate, retry count, cumulative waiting time, final failure rate, and queue depth. A successful retry policy should recover occasional throttling without hiding sustained overload. If retries keep rising, reduce admission pressure or investigate capacity instead of simply extending the wait.