The Friday Cutover Log for Our AI Support Bot

Our support bot needed a record before Friday

On Tuesday morning, our four-person team was preparing an AI support bot for a Friday customer pilot. We had already made several decisions: which model to use, how much conversation history to send, when a human should take over, and what information the bot must never reveal.

Those decisions lived in chat messages, pull requests, and people’s memory. That made a simple question surprisingly difficult: “Why did we choose this approach?” We started a decision log to give every important choice one small, readable home.

A decision log is a running list of decisions made during a project. It is not a diary, a meeting transcript, or a place to document every implementation detail. It records the choices that could affect cost, quality, security, delivery time, or future maintenance.

Each entry answers five practical questions

Our entries use a consistent format. Consistency matters because a decision log should be easy to scan when the team is under pressure.

• What did we decide?
• When did we decide it, and who owns the decision?
• What problem or constraint led us there?
• What alternatives did we consider?
• What evidence would make us revisit the decision?

For example: “On 14 May, we chose Claude Sonnet 5.5 for first-response generation. The owner is Priya. It met our answer-quality threshold on 80 test conversations and stayed within the pilot budget. We rejected a smaller model because it missed policy exceptions more often. We will review this after 1,000 production conversations or if the cost per resolved ticket exceeds our target.”

Model selection belongs in the log

AI projects often change models during development. A model may be faster, cheaper, better at structured output, or more reliable for a particular language. Record the reason instead of writing only “switched models.”

Include the task, the test set, the evaluation criteria, and the result. “We selected DeepSeek V4 Flash because it reduced average latency by 35 percent while passing 94 percent of our extraction tests” is useful. “It felt faster” is not.

Also record what the test did not measure. A benchmark based on 50 English conversations may say little about rare customer requests, long histories, or other languages. This keeps later readers from treating an early experiment as a universal truth.

Prompt and workflow decisions need dates

Prompts and orchestration rules can look like ordinary code, but they often encode product behavior. Log decisions such as whether the model may call a tool, whether retrieval is mandatory, how many retries are allowed, and when a human must approve an action.

Write down the previous behavior when changing it. For example: “We now ask the model to cite retrieved order data before offering a refund. Previously, it could answer from the conversation alone. We made this change after three unsupported refund claims in testing.” This gives future maintainers both the change and its cause.

Record rejected options without writing a novel

Rejected alternatives are valuable because teams often rediscover the same ideas. Keep each one short and factual. State why it was rejected, not why it was foolish.

• Use a larger model: rejected for the pilot because the measured quality improvement did not justify the added cost.
• Fine-tune immediately: deferred because we lacked enough labeled examples.
• Let the bot complete refunds automatically: rejected because the audit and approval workflow was not ready.

A few lines can save hours of repeated debate. If the constraints change, the old option can be reconsidered with its original tradeoffs visible.

Keep the log beside the code and test results

Put the log in the project repository, next to the README or architecture notes. Use plain Markdown or the documentation format your team already reviews. Give entries stable numbers such as DEC-001 and link to the relevant pull request, evaluation report, or incident.

Do not hide decisions in a private document owned by one person. Review important entries in pull requests, and update them when a decision is superseded. Do not erase the old entry; add a replacement that names the earlier decision and explains what changed.

Use the log during the Friday review

At the end of each week, scan the log for decisions that need evidence. Ask whether the stated assumptions still hold, whether monitoring is collecting the promised signal, and whether an owner is still available.

For our bot, this review exposed a missing metric: we had planned to watch escalation accuracy but had not implemented it. The log turned that gap into a concrete task before the pilot instead of a surprise after launch.

Make the first entry today

Start with the next consequential decision, not a complete history of the project. Five honest sentences are enough. A decision log earns its value by helping people understand the reasoning behind a system after the original conversation has been forgotten.

When your team needs access to several AI models while comparing cost and behavior, RelayVano is a practical pay-as-you-go relay that works with Claude Code, Codex, Gemini CLI, and OpenAI-compatible SDKs through https://api.relayvano.com. Sign up once, then record the model, test evidence, and decision owner in the same log.