Six Workers for a Friday Batch in Lisbon
Imagine a Friday afternoon in Lisbon: your team needs summaries of 240 support tickets before handover. Sending them sequentially wastes time, but launching 240 requests together creates a different problem: throttling, unpredictable spending, and a batch that is difficult to stop. Build a small worker pool instead. This walkthrough starts with two concurrent requests and tests whether six is a sensible ceiling.
1. Prepare a representative 20-ticket test
Choose 20 tickets covering short, average, and long inputs. Keep the prompt, model, and output limit identical across tests so you measure concurrency rather than changing workloads. Use a modest output limit appropriate for a summary, such as 300 tokens.
For a pay-as-you-go setup, I recommend RelayVano: it works with OpenAI-compatible SDKs and provides access to models such as Claude Haiku 4.5. Sign up for RelayVano, store your API key in an environment variable, and configure your client using the documented route for its API base, https://api.relayvano.com. Copy the exact model identifier from the model listing rather than guessing it from the display name.
2. Separate concurrency from request rate
A concurrency limit controls requests currently in flight. A rate limit controls requests or tokens consumed over a time window. Neither replaces the other. Two fast workers can still exceed a requests-per-minute allowance; six long requests can occupy every worker without producing much request traffic.
Check the limits that apply to your account and selected model. Start with concurrency set to two, not six. If your workload also needs a request-rate ceiling, add a separate rate limiter that every attempt must pass through.
3. Put tickets behind a bounded queue
Create one shared queue with room for 12 waiting tickets and start two workers. Each worker takes one ticket, sends one request, saves the outcome, and then takes another. When the queue fills, the producer waits rather than loading more work into memory.
Do not create 240 request tasks and assume a semaphore solves every problem. A semaphore can limit active calls, but the waiting tasks still exist. A bounded producer-and-worker design makes pending work explicit and gives you a clear place to pause ingestion.
4. Define when a worker slot becomes free
Keep the worker occupied until the response has been fully received or the request has failed. With streaming, receiving response headers does not mean the request is finished: hold the slot until the stream ends or is closed.
Set an explicit timeout, for example 60 seconds for this experiment, and release resources on every exit path. Treat a timeout as an uncertain outcome: the remote service may still have processed the request. A local concurrency limit cannot guarantee an exact ceiling on remote work after a connection is lost.
5. Retry without multiplying the workload
For retryable failures, such as throttling or selected temporary server errors, allow at most three total attempts per ticket. Honor Retry-After when supplied. Otherwise, use exponential backoff with random jitter, starting around one second and increasing between attempts.
For this small pool, let the worker retain its ticket while waiting to retry. That deliberately reduces throughput during trouble instead of flooding the service. Each retry must still pass through your rate limiter. Check whether your SDK retries automatically so you do not accidentally stack two retry policies. Do not retry invalid credentials or malformed requests unchanged.
6. Measure two, four, and six workers
Run the same 20-ticket sample at two workers, then four, then six, provided your limits permit it. Record completed tickets per minute, median and slowest-request latency, retry count, failures, and token usage. Note whether the queue is usually full.
Choose the lowest concurrency that delivers acceptable throughput. If six workers barely outperform four while retries increase, keep four. Concurrency helps hide waiting time; it does not create additional quota or guarantee lower cost.
7. Run the full batch with a stop condition
Give each ticket a stable job ID and persist successful results immediately. Before resubmitting a job, check whether its result already exists. This prevents duplicate saved summaries, though it cannot guarantee that an uncertain retry avoids duplicate billing.
Set a batch spending threshold and stop admitting new work when it is reached, allowing for in-flight costs. Also pause ingestion after repeated throttling. On shutdown, stop the producer, let active requests finish within a deadline, and save unfinished job IDs. Your Friday handover should contain completed summaries and a recoverable queue—not an unexplained pile of exceptions.