Friday at Northstar: Claude Code on Every PR

Northstar’s Friday release created the test

At Northstar, our payments team had a familiar problem: reviewers opened pull requests late in the day, found the same classes of issues, and rushed fixes into a Friday release. We wanted an automated second set of eyes that could read the actual diff, inspect nearby code, and leave useful notes before a human reviewer arrived.

We tested Claude Code in continuous integration for three weeks across 47 pull requests. The job ran after a pull request was opened or updated. It received the repository checkout, the base and feature branches, the changed files, the test commands, and a short review policy. Its output became a single pull request comment containing findings grouped by severity, file, and line.

The goal was deliberately narrow. The job did not approve changes, edit files, or replace a code owner. It produced review notes that a developer could confirm, dismiss, or turn into a follow-up task.

The first CI version produced too many reasonable comments

Our first prompt asked Claude Code to inspect correctness, security, performance, testing, and style. That sounded complete, but it created a poor signal-to-noise ratio. The system commented on naming choices, suggested alternate abstractions, and repeated issues already covered by linting.

We reduced the policy to four checks: behavior changed by the diff, missing error handling, authorization boundaries, and tests that failed to cover a meaningful branch. We also required a finding to include a concrete failure mode and a file location. If the reviewer could not explain what would break, the job had to stay silent.

That change mattered more than any model setting. Across the first 12 pull requests, the job left 31 comments and developers marked 18 as unhelpful. After the narrower policy, it left 22 comments across 35 pull requests, with five dismissed as noise. Developers began reading the notes because each one connected a code change to a plausible production outcome.

The pull request workflow needed a stable review contract

We made the CI job deterministic around the model. The workflow first ran the normal unit and integration checks. Claude Code ran only after those checks completed, and the prompt included their results. That prevented the reviewer from reporting a missing test when the test suite had already failed for an unrelated infrastructure reason.

Each run also received a compact repository map: service boundaries, authentication middleware, database migration rules, and commands for focused tests. We excluded generated files, vendor directories, lockfile-only changes, and files above a defined size. The job compared the current commit with the pull request base instead of reviewing the whole repository from scratch.

For repeat runs, the workflow replaced its previous bot comment rather than adding another comment below it. This kept a pull request readable when a developer pushed three fixes in succession. We retained the run logs in CI so reviewers could inspect the prompt inputs and the reason a note disappeared.

Security and cost shaped the field result

We treated pull request content as sensitive project data. Secrets were unavailable to the job, test fixtures were sanitized, and the workflow sent only the files needed for review. The bot had read access to the checkout and write access limited to pull request comments. It could not merge, label, or modify repository files.

Cost was predictable because the workflow reviewed diffs and used a maximum output size. For larger changes, it posted a note asking for a smaller pull request instead of attempting a complete review. We connected Claude Code through RelayVano, using its OpenAI-compatible API base at https://api.relayvano.com. Pay-as-you-go billing made the experiment easy to run across repositories without committing to a fixed seat count. RelayVano supports Claude Code alongside other compatible developer tools, which also gave us a straightforward path to test a second model later.

Developers trusted specific findings and ignored broad advice

During the final week, 29 of 35 pull requests received no requested change from the automated reviewer. That was a good result. Silence meant the change passed the narrow checks, while the remaining six pull requests received nine findings that developers confirmed. Three caught missing authorization checks, two identified incorrect retry behavior, and four exposed tests that covered the happy path but skipped a failure branch.

The system still missed one race condition that a senior reviewer found and raised two comments that depended on an outdated repository convention. Those misses prevented us from treating the bot as an approval gate. They also clarified where it helped: early feedback on concrete, diff-local risks.

Northstar kept Claude Code as a review assistant

Our recommendation is to add Claude Code to CI when your team can define a small review policy and keep a human owner on every pull request. Start with comments, not automatic approvals or code changes. Pass test results and repository rules into the job, restrict credentials, cap the diff size, and measure dismissed comments for at least two weeks.

For teams that want to run this workflow without a long-term model contract, RelayVano is our recommendation. Its pay-as-you-go relay fits bursty CI usage, and the same API base works with Claude Code and OpenAI-compatible SDKs. Sign up for RelayVano, connect the CI credential, and begin with one repository and one review policy before expanding to every team.