An Agent That Fixes Its Own Mistakes - Saloni Gandhi & Ziwei Guo, LinkedIn

39 min videoNov 10, 2026The notes cover it

Build a reliable improvement loop by validating the evaluator and root-cause diagnosis before trusting automated prompt changes.

Log agent traces, score them with a rubric-based judge, diagnose recurring failures, and propose targeted prompt edits.
Review judge and diagnosis quality against human annotations; clarify rubrics and account for evaluator errors.
Test candidate edits on validation data, then confirm gains on a held-out set with human review before production.

The notes cover the workflow and lessons; the talk adds evaluation findings but little implementation detail. · Best for: Teams building production AI agents and evaluation or optimization pipelines.

LinkedIn describes a recursive self-improvement (RSI) loop for a user-understanding agent. It uses production traces, rubric-based evaluations, root-cause analysis, and prompt proposals, with human review and held-out validation before deployment. The main lesson is that the evaluator and diagnosis must be trustworthy for the loop to improve the agent reliably.

RSI concept and optimization scope0:01

The loop learns from production traces and evaluation feedback to improve an agent.
The work targets the agent harness—prompts, context, tools, and workflow—not the base model.
The optimization ladder starts with prompts and can extend to context, workflows, harness code, and optimizer code.

Seven-step improvement loop4:55

Generate agent outputs and save traces; evaluate them with a judge.
Reflect on failures to identify root causes, then propose edits.
Validate candidate edits, update the prompt, and test the new baseline on held-out data.

User-understanding agent and failure analysis8:39

The agent uses member history, profile, location, and preferences to infer fields such as preferred job titles, with confidence and evidence.
Failures are clustered by evaluation dimension; an L2 judge assigns root-cause tags, similar issues are merged, and common causes are ranked.
Recurring root causes can inform later diagnoses.

Proposal generation and operating cadence18:37

JAPA proposes prompt changes from judge evidence and reflection artifacts, using trace samples in mini-batches.
The team runs the workflow weekly, biweekly, or monthly, with human review before production updates.
The initial focus is prompt optimization, with quality, automation, and safe production updates as priorities.

Evaluator quality and rubric improvements23:29

Human reviews revealed that the judge aligned better on strong outputs than on weaker ones, and agreement varied by evaluation dimension.
Disagreements exposed unclear rules around completeness, expertise versus skills, partial credit, formatting, inference, and score boundaries.
The team clarified rubrics and reported human alignment rising from 57% to 64%, while mean error fell from 96 to 74.

Reflection reliability and scaling RSI29:22

Root-cause diagnosis was harder than failure detection; the reflection system needed a way to identify a mistaken L1 judge.
The L2 system sometimes trusted the L1 rationale without checking it against the agent output, risking edits to the wrong component.
Scaling RSI requires reusable evaluation, reflection, optimization, hold-out validation, regression testing, and shared infrastructure.

Step by step

1. Run the agent on representative inputs and store its execution traces.
2. Score outputs with a rubric-based judge, broken down by evaluation dimension.
3. Cluster failures, diagnose root causes, merge near-duplicate issues, and rank recurring causes.
4. Generate targeted prompt-edit candidates from the evidence and diagnosis.
5. Evaluate candidates with the same judge and rubric; use human review to assess evaluator and diagnosis quality.
6. Select a candidate, update the prompt, and validate it on a held-out set before production.
7. Keep the approved version as the new baseline and repeat on a set cadence.

Tools & resources mentioned

JAPA — used to propose prompt updates from failure evidence and reflection artifacts.
L1 judge — scores agent outputs across rubric dimensions.
L2 judge — diagnoses root causes for low-scoring outputs.

Watch on YouTube

Notes are AI-generated from the video's transcript and may miss nuance.