Your PR Says the Tests Passed. Prove It. - Stephen Chin & Cassandra Chin
Build an evidence graph from agent traces and code relationships so PRs make verifiable claims with fewer review passes and lower token use.
- Use agent-session traces, code, and test records to support PR claims instead of relying on a compressed chat context.
- Build a connected evidence packet before drafting; use cheaper models for mechanical review and a stronger model for writing.
- Benchmark PR quality and token use against your current workflow before adopting the approach.
The core workflow and findings are covered in the notes; the talk mainly illustrates them with a case study. · Best for: Teams using coding agents that need more accurate PRs and lower review costs.
The speakers describe how PR descriptions generated from an agent's active session can contain inaccurate test claims or omit evidence. Their approach uses Trace Tune and Relate to collect traces, link relevant facts, and give a writer and reviewers a compact evidence packet. They report better benchmark scores and about half the previous token use for their project.
Why agent-written PRs can be unreliable0:00
- PRs may be rejected because their descriptions contain incorrect information, even when the code is useful.
- A coding agent may infer what happened from its session context, which can become compressed and less accurate.
- Important work such as prompts, commands, and test runs may not be visible in the submitted code.
Trace Tune gathers context for PRs6:48
- Trace Tune finds traces for a branch, researches them, reviews the draft, and repeats until it has supporting evidence.
- The goal is to explain the work and evidence, not merely describe the code diff line by line.
- The speakers say the initial workflow improved PRs but could still make mistakes.
Evidence graphs address accuracy and cost9:10
- A test claim was wrong because the trace showed two commands run concurrently even though they needed to run sequentially.
- Simply switching to cheaper writer models reduced cost but also lowered writing quality.
- Relate stores relationships locally and builds an evidence packet without LLM costs for that graph work.
Use a connected brief and assign models by task12:45
- A neighborhood brief connects code changes, checks, attributions, and their sources into one packet.
- The speakers use cheaper models for mechanical review and cross-checking, while retaining a stronger model for the final PR writing.
- The example brief distinguishes what passed from what was not verified before the change.
Benchmark results and reported savings15:58
- PRBench compares results using audit traces, code, model judges, and human review.
- The speakers report score changes from 74 to 76 and 82 to 86 in their comparisons.
- They report that Trace Tune with Relate used about half as many tokens as the earlier workflow.
Where the approach may apply18:19
- The speakers suggest using a custom graph layer when a project has related data, evidence, or items that need comparison.
- They caution that agents are not as good at data analysis, evaluation, and curation as they are when given organized context.
Step by step
- Collect the agent traces, code changes, and test records associated with the branch.
- Create a graph that links relevant changes, checks, command context, and sources.
- Generate a compact evidence brief from connected facts before drafting the PR.
- Use lower-cost models to check claims and identify missing findings; use a stronger model to write the final description.
- Compare PR accuracy and token use with your existing process using a benchmark and human review.
Tools & resources mentioned
- Trace Tune — open-source PR-writing workflow that uses agent traces and code context.
- Relate — open-source, embedded Rust graph database used to build evidence relationships.
- PRBench — benchmark the speakers used to compare PR-description quality.
Notes are AI-generated from the video's transcript and may miss nuance.