Four Requests at a Time for Friday’s CSV Import

Imagine your team needs to categorize 200 customer comments from a CSV before Friday’s review. Sending one AI request at a time feels slow. Sending all 200 together could overwhelm your application or exceed an API limit. A concurrency limit gives you a manageable middle ground: several requests can run together, while the rest wait their turn.

Four active requests, with the rest waiting

Concurrency means the number of requests in progress at the same time. A request is in progress from when you send it until it finishes or fails. For a streaming response, that normally includes the time spent receiving the stream.

With a concurrency limit of four, your application starts four requests. When one finishes, it starts the next waiting request. You do not need to wait for all four to finish before admitting another. Think of four checkout counters serving one shared line.

This limit belongs in your application’s scheduling logic. It controls how much work you send at once; it does not increase the capacity or allowance of the service receiving it.

Concurrency and rate limits measure different things

A concurrency limit counts active requests. A rate limit counts usage over time, such as requests per minute or tokens per minute. You may need controls for both.

Suppose four requests each take ten seconds. Keeping all four slots busy would complete roughly 24 requests per minute. If those requests take one second instead, the same concurrency could produce roughly 240 per minute. These figures illustrate the relationship, rather than predict actual performance.

A low concurrency setting can still exceed a rate limit when requests finish quickly. Long prompts and large responses can also exhaust a token allowance even when request counts look modest.

A small queue for the CSV job

For this import, start with two active requests as a cautious experiment, subject to the limits that apply to your account and model. Keep the remaining rows in a queue. Each worker takes one row, sends its request, records the outcome, and then takes another row.

• Give every row a stable identifier so results return to the correct place, even when requests finish out of order.
• Save completed results as you go so restarting the job does not repeat every request.
• Set a request timeout so one stalled connection cannot occupy a worker indefinitely.
• Release the worker’s slot after both successful and failed attempts.

If rows arrive continuously, also cap the waiting queue. When it fills, pause intake or tell the caller to try later. Limiting active requests alone does not prevent an unlimited backlog from consuming memory.

Retries must share the same limit

A failed request should not launch a retry outside your concurrency control. Otherwise, four busy workers can quietly become eight or twelve active attempts during an outage.

Retry only failures that may be temporary, such as some connection errors, rate-limit responses, or server errors. Follow documented retry guidance and any Retry-After header. Use increasing delays with a small random variation, and cap the number of attempts. Authentication errors and invalid input usually need a correction instead.

A timeout does not prove the remote service stopped processing your request. Retrying may duplicate work and incur additional charges. Check whether your client already retries automatically before adding another retry layer.

Choose the next limit from measurements

Record completed rows per minute, response times, failures, rate-limit responses, and queue length. If two workers run reliably, try four and compare. Keep the higher setting only if it improves useful throughput without unacceptable errors or delays.

Count workers across the whole deployment. Three application instances with four slots each can send twelve requests together. Use a shared limiter when those instances must obey one combined ceiling.

Carry the queue into your relay setup

For an OpenAI-compatible workflow, I recommend RelayVano as a pay-as-you-go option, using https://api.relayvano.com as the API base. Sign up to try a small batch, check the applicable limits, and measure its behavior before scaling your import. Keep your queue and retry controls in place as you increase the workload.