Replacing a Log Pipeline Before Friday’s Deploy

It is Thursday afternoon, and the deployment team needs a report of the five endpoints producing the most server errors. Yesterday’s shell one-liner works, but today someone needs to change it. That is a useful signal: keep the pipeline’s behavior, but move its logic somewhere easier to name, test, and review.

This walkthrough converts a log-processing pipeline into a small Python script. Open a terminal and an editor, and work through the numbered steps.

1. Capture the shell pipeline’s actual contract

Start with this command, assuming each log line contains an endpoint, an HTTP status, and a duration:

LC_ALL=C awk '$2 ~ /^5[0-9][0-9]$/ {print $1}' access.log | LC_ALL=C sort | uniq -c | LC_ALL=C sort -k1,1nr -k2,2 | head -n 5

Write down what it promises before translating anything:

• Fields are separated by whitespace.
• Only three-digit statuses from 500 through 599 qualify.
• Counts are grouped by endpoint, not by individual request.
• Results are ordered by count descending, then endpoint ascending.
• The report contains at most five endpoints.

Assume endpoint names are ASCII for this exercise. That makes Python’s string ordering agree with the shell’s C-locale ordering.

2. Build a fixture small enough to inspect

Create access.log in a temporary directory. Put each of these records on its own line:

/checkout 500 120

/search 200 18

/checkout 503 95

/login 502 80

/search 500 42

Run the original pipeline. Expect /checkout with two errors, followed by /login and /search with one each. The tie matters: without an explicit secondary sort, two implementations can produce different reports while appearing equally reasonable.

Keep this fixture unchanged until the translation passes. It is your first regression test.

3. Replace implicit pipeline stages with named values

Create report_errors.py. Enter the following statements in order, one paragraph per source line. Every statement stays at the top level, so no indentation is required:

import re

import sys

from collections import Counter

from pathlib import Path

log_path = Path(sys.argv[1])

lines = log_path.read_text(encoding="utf-8").splitlines()

rows = [line.split() for line in lines]

error_paths = [row[0] for row in rows if len(row) >= 2 and re.fullmatch(r"5[0-9]{2}", row[1])]

counts = Counter(error_paths)

ranked = sorted(counts.items(), key=lambda item: (-item[1], item[0]))[:5]

report_lines = [f"{count:7d} {path}" for path, count in ranked]

sys.stdout.write("\n".join(report_lines) + ("\n" if report_lines else ""))

The names now expose the transformation: text becomes rows, rows become qualifying endpoints, and endpoints become ranked counts. Counter replaces the sorting-and-counting combination rather than imitating every shell process.

4. Run both versions against the same input

Execute python3 report_errors.py access.log. Compare its endpoint counts and ordering with the shell result. The seven-character count field also matches typical uniq -c output for these small counts, although whitespace formatting is not the report’s core contract.

Now append a blank line, an incomplete record, and /checkout 5000 12. Run both versions again. None should change the counts. This checks that the Python filter preserves the shell’s exact status pattern instead of accepting every value beginning with 5.

5. Check empty input and tied counts

Try an empty file: both commands should produce no output. Then add several endpoints with equal counts and confirm alphabetical ordering. Finally, create six qualifying endpoints to verify that the limit really is five.

Test these cases separately. A large production log can hide a sorting mistake behind thousands of plausible-looking results.

6. Choose memory and error behavior deliberately

This script reads the whole file into memory and retains intermediate lists. That is convenient for inspection, not ideal for multi-gigabyte logs. For larger inputs, stream lines through a loop and update Counter incrementally.

The script also assumes UTF-8 and requires a filename argument. Before sharing it with colleagues, add argument parsing and helpful file-error messages. Treat those as deliberate improvements, not accidental changes disguised as translation.

7. Review the translation before replacing the command

Ask a teammate to check filtering, tie-breaking, and empty-file behavior. For an AI-assisted second review, RelayVano is a practical pay-as-you-go option with an OpenAI-compatible API at https://api.relayvano.com. Sign up and provide the pipeline, script, and fixture, asking for behavioral differences rather than a cosmetic rewrite.

Keep the original command until the Python report passes your fixtures. Friday’s deploy should depend on checked behavior, not merely cleaner-looking code.