I Let Agents Refactor My Codebase for 3 Weeks. Then I Read the Code. — Keiji Kanazawa, Microsoft
Own the goal and define outcome rubrics; use agents to iterate against them, but inspect results and have agents review each other.
- Start fresh when a refactor obscures the system’s real behavior; write clearer specs and acceptance rubrics.
- Evaluate the output, revise the rubric, and test on varied examples to prevent overfitting to one case.
- Run adversarial reviews between coding agents for several rounds, and check tests do not call production inference.
The video demonstrates a rubric-driven workflow and a video-analysis canvas, beyond the summary. · Best for: Product managers and developers building agent workflows with coding agents.
After three weeks of agent-led refactoring, the speaker found misleading abstractions, broken workflows, and tests consuming production tokens. He restarted with clearer specs, outcome rubrics, iterative evaluation, and cross-agent reviews, while continuing to delegate most coding.
From deterministic workflows to agents1:47
- A personal system for summarizing podcasts and links grew into an LLM wiki and scheduled agent workflows.
- The speaker tasked Codex and Claude with implementing and reviewing the system.
Refactor failures surface4:09
- A misattributed digest entry prompted an investigation; the system lacked the agent traces needed for the desired analysis.
- Skills, actions, and agents had names that implied capabilities they did not actually have.
Cost and broken workflows6:23
- Codex connected unit tests to production inference, running about 30 test cycles and using production tokens.
- A daily reflection agent produced empty results after the refactor broke its prompts.
Restart and compare frameworks7:26
- The speaker stopped trusting the existing implementation, started a fresh version, and wrote new specs and rubrics.
- He tried four frameworks and settled on Claude Code and Codex for personal projects; he liked Pi’s result but found its design questions too engineer-focused.
Build a video-analysis canvas12:06
- The speaker’s video-dive workflow selects key frames, uses an evaluation rubric, and presents explanations in an explorable canvas.
- The canvas organizes chapters and equations, links back to video sections, and supports questions about the material.
Iterate on rubrics and cross-review14:38
- Agents drafted rubrics from the speaker’s directions; he revised them based on the generated canvas.
- Testing on multiple videos exposed a rubric overfit to one lecture; correcting it improved generalization.
- Claude and Codex reviewed each other’s code through multiple adversarial review rounds.
Step by step
- Write a fresh spec that defines the desired system and its behavior.
- Add an outcome rubric describing what good results should look like.
- Generate an output, inspect it, and revise the rubric where it falls short.
- Test the rubric across varied examples to catch overfitting; repeat the evaluation loop.
- Have separate coding agents review each other’s changes for several rounds.
- Verify tests use safe test inference rather than production inference.
Tools & resources mentioned
- Claude Code — coding agent used for implementation and review.
- Codex — coding agent used for implementation and review.
- GitHub Copilot CLI — coding tool the speaker uses at work.
- Pi — agent framework tested in the framework comparison.
- YouTube — source videos for the speaker’s video-analysis workflow.
Notes are AI-generated from the video's transcript and may miss nuance.