A 429 During Friday's Model Launch in Toronto

At 4:42 p.m. on Friday, a team in Toronto is testing its new AI-powered support flow before a Monday launch. Most requests work. Then the logs begin filling with 429 responses, and the test runner keeps sending requests as fast as before. The result is more failures, not faster progress.

A 429 means the server is asking your application to slow down. It has received too many requests for a limit that applies to your account, model, or time window. The response is not necessarily evidence that your request is malformed. It is a signal to pause, then try again in a controlled way.

What a 429 rate limit means

Services set rate limits to keep shared capacity reliable. A limit might count requests per minute, tokens per minute, or simultaneous requests. You can stay below one limit and still exceed another: for example, a few long prompts may use a token allowance quickly, even when your request count is low.

Check the response body and headers for details. Some APIs include a Retry-After value, which tells you how many seconds to wait or, in some cases, the time at which to retry. Treat that instruction as useful guidance. Your account dashboard or provider documentation may also describe the relevant limits.

Why exponential backoff gives the service room

Exponential backoff means waiting longer after each consecutive failure. Instead of retrying immediately, an application might wait one second, then two, then four, then eight. If the next attempt succeeds, the retry sequence ends. If it fails again with a retryable response, the wait grows.

This gives the service time to recover and prevents many clients from hammering it at once. Add a small random variation, called jitter, to each wait. Without jitter, clients that fail together may retry together and cause another burst. For example, a delay near four seconds could be randomized to somewhere between three and five seconds.

What to retry after a 429

Retry only when the response indicates a temporary limit. Honor Retry-After when it is present, and do not retry sooner than the requested delay. When it is absent, use a capped exponential schedule with jitter. Set a maximum number of attempts or a total time budget, such as five tries over one minute. After that, return a clear error or queue the work for later.

Do not apply the same policy indiscriminately to every failure. A 401 usually needs corrected credentials, and a 400 usually needs a corrected request. Repeating either unchanged is unlikely to help. For network timeouts, consider whether the original request may already have completed before sending it again; duplicate work can matter for operations with side effects.

How to keep retries from becoming a second problem

Retries should be visible in logs and metrics. Record the status code, attempt number, wait duration, and model or route, while avoiding sensitive prompt content. Alert on sustained 429 rates rather than a single event. This helps distinguish a brief burst from a workload that regularly exceeds its allowance.

Also limit concurrency and smooth out bursts. A queue can release requests at a steady pace, while a semaphore can cap how many calls run simultaneously. If an SDK already retries automatically, check its settings before adding another retry loop. Nested retry policies can multiply attempts and make a brief limit issue much worse.

Applying the pattern to AI model calls

In an AI application, make retry behavior part of the client layer so each feature follows the same rules. Keep prompts and request identifiers available for safe diagnosis, and give the caller a useful outcome when the retry budget is exhausted. For workloads that can wait, place failed jobs back in a queue rather than keeping a user-facing request open indefinitely.

RelayVano provides pay-as-you-go access to models including Claude Sonnet 5.5 and DeepSeek V4 Flash, and works with Claude Code, Codex, Gemini CLI, and OpenAI-compatible SDKs through https://api.relayvano.com. If you need a relay for testing or running AI workflows across providers, sign up for RelayVano and apply the same measured retry policy in your client.