60% of Our Data PRs Now Merge With No Human Review - Chetas Joshi, Robinhood
Use scripts for verifiable facts, an agent for contextual risk, and humans for sensitive or uncertain decisions.
- Run deterministic CI checks against authoritative sources for schemas, ownership, PII tags, lineage, and performance.
- Have the agent search repository code for unregistered consumers, explain risks, and escalate when confidence is low.
- Keep sensitive-data grants and changes to data meaning with human reviewers; track both misses and false escalations.
The walkthrough shows how layered checks catch a real downstream break and where humans remain essential. · Best for: Data platform teams handling high volumes of production pipeline PRs
Robinhood describes a hybrid AI reviewer for production data-pipeline pull requests. Scripts check facts against live sources of truth, an agent searches repository code for contextual risks, and people retain protected decisions. The speaker reports that about 60% of these PRs merge without human review, while emphasizing that human accountability remains.
The review bottleneck and hybrid approach0:00
- AI assistants make data-pipeline PRs arrive faster than a small group of reviewers can assess them.
- The system assigns deterministic checks to scripts, contextual judgment to an agent, and protected calls to people.
- A merge requires passing checks and an approving review.
Layer 1: deterministic checks6:05
- Compare schema changes with the schema registry; check table ownership and PII labels in the catalog.
- Walk registered lineage to find affected consumers and run a staging performance test at production data volume.
- These checks only know what their source systems record; passing them does not prove all downstream use is safe.
Layer 2: agent review of repository code9:48
- The agent reads, searches, and lists repository files; it cannot query live catalog, lineage, or production state.
- It looks for query anti-patterns, risky grants, and consumers missing from registered lineage.
- It must say where it searched; finding no consumers is not proof that none exist.
Risk findings and human escalation14:28
- The agent provides line-specific comments and reasoning, scores risk, and either approves or escalates.
- In the example, it found an unregistered job that would write a widened integer value into an integer column.
- A human confirmed the job was active; the downstream table was updated before the PR merged.
Human-only decisions and feedback16:54
- Sensitive-data grants, changes to data meaning, and low-confidence verdicts always go to a person.
- People explain agreement or disagreement; the on-call reviewer uses that feedback to update rules weekly.
- An example showed that encryption before writing affected how a PII label was assessed, but this was not treated as a blanket rule.
Measuring performance and improving rules20:24
- Track bad PRs approved (misses) and safe PRs escalated (false escalations).
- The speaker reports an approximately 2% miss rate as of August 31, 2026; no single false-escalation rate was available.
- A missed downstream dependency after sensitive retagging led to a new rule to search for readers of the copy losing those columns.
Results, limitations, and lessons22:44
- Reported results: about 60% of data-pipeline PRs merged without human review, 75% less manual review effort, 132 engineering hours saved monthly, and 30% fewer related incidents.
- The current script and agent systems run independently; the agent cannot access live checks, and there is no automatic fallback if a check cannot finish.
- A unified entry point and callable layer-one checks are proposed, not yet built.
Step by step
- Run deterministic CI checks against authoritative systems for schema contracts, ownership and PII tags, registered lineage, and performance.
- Have the agent inspect the PR and relevant repository files for contextual risks, including unregistered consumers.
- Require the agent to state its search limits, explain findings, and assign a risk level.
- Approve only when required checks pass and either the agent or a human approves.
- Always route sensitive-data grants, changes to data meaning, and low-confidence cases to a human.
- Collect human explanations, review feedback weekly, update rules, and measure both misses and false escalations.
Tools & resources mentioned
- Schema registry — source of truth for schema and contract checks.
- Data catalog — checks table ownership and PII labels.
- Lineage tool — identifies registered downstream consumers.
- Repository file tools — let the agent read, search, and list code and configuration.
Notes are AI-generated from the video's transcript and may miss nuance.