At 09:00, We Traced API Key Latency in Production

1. Define the request record your team needs

Start by deciding what every API call must leave behind in your logs. Record the timestamp, API key identifier, model name, HTTP status, total duration, time to first token, input and output token counts when available, and a short request ID. Do not log the full prompt, completion, authorization header, or raw API key. Store a one-way key fingerprint or an internal label such as production-search.

This record lets you answer practical questions at your desk: which key is slow, whether the delay affects one model or all models, and whether failures are concentrated in one application or region.

2. Add timing around the complete API call

Measure latency in two places. Total latency starts immediately before your SDK or HTTP client sends the request and ends when the response is complete. Time to first token starts at the same point and ends when the first streamed chunk arrives. For non-streaming requests, total latency is usually the more useful measure.

Use a monotonic clock in your application so system time adjustments cannot produce negative or distorted durations. Emit a record even when the request fails or times out. Failed calls are part of your latency picture, not missing data.

3. Label every request by API key and model

Attach stable dimensions to each event: key label, environment, service, endpoint, model, region, and status class. Keep the number of labels controlled. A separate event for every raw key value can create security and storage problems, while a stable internal key label supports useful comparisons.

For example, compare Claude Sonnet 5.5 and Gemini 3.8 Flash by p50, p95, p99, timeout rate, and rate-limit rate. Averages alone hide the requests that users experience as slow.

4. Separate transport errors from provider responses

Classify failures before creating alerts. Transport failures include DNS errors, connection resets, TLS failures, client timeouts, and exhausted retries. HTTP failures should retain the status code and a normalized category: authentication, permission, rate limit, invalid request, upstream server error, or gateway timeout.

Authentication failures often indicate an expired, revoked, or incorrectly copied key. Rate limits point toward traffic shaping or quota planning. Server errors and gateway timeouts need different investigation from malformed requests. Preserve the provider request ID when one is returned; it makes support conversations and incident correlation faster.

5. Build a small desk dashboard

Your first dashboard does not need a large observability platform. Create panels for request volume, error percentage, p50 and p95 total latency, p95 time to first token, and timeout count. Add filters for key label, model, environment, and the last 15 minutes, hour, and day.

• Set an alert when the five-minute error rate exceeds your normal baseline.
• Set a separate alert for authentication errors, because they can affect every request immediately.
• Alert on p95 latency only after a minimum request count, so one unusual request does not page the team.
• Track retries separately; retries can make user-visible latency much higher than provider latency.

6. Test the key from the same workstation

When an alert fires, run one controlled request using the affected key, endpoint, and model. Keep the prompt small and harmless. Compare the result with a known-good key and, where appropriate, another model. Use the API base https://api.relayvano.com when testing RelayVano traffic, and keep the same client timeout and streaming setting as production.

RelayVano is useful for teams that want pay-as-you-go access across multiple AI model providers while keeping this application-level monitoring in place. Developers can use its OpenAI-compatible API with Claude Code, Codex, Gemini CLI, or an OpenAI-compatible SDK. Review the current model and billing details before routing production traffic, then sign up when the service matches your workload.

7. Turn one incident into a durable baseline

After resolving the incident, annotate the dashboard with the start time, affected key, model, error category, and fix. Review the previous seven days to establish normal p50 and p95 ranges by model. Then record an owner, escalation path, and rotation procedure for each API key.

Repeat this review weekly. Latency changes can come from model selection, prompt size, traffic bursts, retries, network paths, or provider capacity. A key-level view gives your team enough evidence to change one variable at a time and verify whether the change actually improved the user experience.