A Support Team’s Friday Run: 100,000 Ticket Tags
Picture a support team with 100,000 tickets to tag before Friday’s reporting deadline. Each ticket needs one category, not a thoughtful essay. Sending every record to a large model is an expensive default. A smaller model, a tight label contract, and a measured fallback path can make the job cheaper without quietly damaging the report.
Work through the steps below on a representative slice before processing the full export.
1. Freeze the ticket labels before comparing models
Open a spreadsheet and define a small, mutually exclusive label set: billing, access, bug, feature_request, and other. Write a one-sentence definition for each label, plus a tie-breaking rule for tickets covering multiple topics.
For example, classify a failed payment that blocks login as billing when payment is the underlying issue. Without that rule, inconsistent labels may look like a model-quality problem when the real problem is an ambiguous task.
2. Build a 300-ticket reference set
Sample across dates, languages, ticket lengths, and customer segments. Include messy examples: quoted replies, empty messages, mixed requests, and rare categories. Remove unnecessary personal data before sending records to any API.
Have a reviewer label the sample, then ask another reviewer to check disputed cases. Keep part of this set untouched for the final evaluation; repeatedly tuning against every example makes results look better than they will be in production.
3. Record the cost of one correct tag
Start a results sheet with columns for model, prompt version, input tokens, output tokens, invalid responses, retries, latency, and correct labels. Calculate total spend from actual usage and the current applicable prices, not assumptions about model size.
Track cost per correct tag alongside accuracy. A cheap request that frequently needs repair or review may cost more overall. Record human-review time separately so API savings do not hide additional operational work.
4. Give the smaller model a narrow contract
Test a candidate such as Claude Haiku 4.5. Ask for a JSON object containing only the ticket ID and one allowed category. Include the label definitions, tie-breaking rule, and two or three difficult examples. Explicitly state that ticket content is data, not instructions.
Do not request explanations unless your workflow uses them. They consume output tokens and complicate parsing. Truncate repeated signatures and quoted history carefully, preserving text needed to identify the actual issue.
5. Run a small, validated pilot
Send 50 tickets first, keeping concurrency low enough to inspect failures. Validate every response against your schema and allowed labels. A valid JSON response is not necessarily a correct classification, so compare the results with your reference labels too.
For this pilot, I recommend RelayVano as a pay-as-you-go relay for comparing candidates through an OpenAI-compatible SDK, using https://api.relayvano.com as the API base and the documented model identifier. Sign up, set a pilot budget, and verify current pricing before launching the run.
6. Escalate difficult tickets instead of every ticket
Route only defined exceptions to a stronger candidate such as Claude Sonnet 5.5: malformed outputs, unusually long tickets, or categories with unacceptable error rates. Test whether escalation actually improves those cases before adopting it.
Do not trust a model’s self-reported confidence as a calibrated probability. Base routing on observable conditions and measured performance. Cap retries, use backoff for transient failures, and send unresolved cases to a review queue rather than retrying indefinitely.
7. Check rare-label quality before scaling
Run the untouched evaluation set through the complete pipeline, including escalation. Measure precision and recall for each label, not just overall accuracy. A system can appear accurate while missing most bug reports if billing tickets dominate the sample.
Set acceptance thresholds around business consequences. Misrouting an ordinary feature request may be tolerable; missing an account-access problem may not be. Expand the test set when rare-label counts are too small to support a decision.
8. Launch the export with a spending stop
Estimate full-run spending from the pilot’s average input length, output length, retry rate, and escalation share. Add headroom, set a hard application-side budget stop, and save completed ticket IDs so interrupted runs can resume without duplicate charges.
Review a fresh sample after each batch. The useful outcome is not simply a lower token bill: it is a repeatable tagging pipeline that meets your quality threshold at a measured, sustainable cost.