Your Coding Agent Can't Undo Its Own Mistakes - Tanmay Sah & Dolly Sah
Before an agent mutates state, capture a witness of the original; then recover and independently verify restoration before accepting the change.
- Treat a timeout as an unknown outcome; check whether an external operation committed before retrying.
- Choose recovery by failure phase: retry before mutation, inspect partial changes, compensate committed effects, or stop and escalate.
- Admit an agent self-change only if it meets capability criteria and an independent verifier confirms recovery.
The key findings and recovery workflow are explained clearly in the notes; the demo is a simple timeout change. · Best for: Teams building coding agents that modify repositories, runtime settings, or external systems.
The talk presents Undo Bench, which measures agent recovery after failures, and EvoUndo, a process for making recoverability a condition of accepting agent changes. Its central point: successful execution does not guarantee safe recovery, especially for effects outside Git.
Recovery failures and unsafe retries0:15
- Undo Bench compared normal execution with injected failures across 2,880 paired trials.
- Agents succeeded on 84% of normal trials; 47% of normally successful trials recovered under failure.
- A lost acknowledgement can hide a completed operation: naive retries caused duplicate effects in 54% of the relevant trials.
- Recovery strategy should depend on whether failure occurred before, during, or after mutation.
Why Git revert is insufficient4:41
- Agents can change runtime configuration, tools, middleware, and routing, as well as trigger deployments or database writes.
- Git revert cannot reverse every effect; recovery must match the effect type.
- EvoUndo records a pre-state witness so the original value can be restored and checked.
- Of 478 capability-improving evolution tasks, 197 had recovery defects.
Grounding and recovery expressivity7:35
- Prompting alone did not repair the naturally occurring recovery failures under the original recovery language.
- Providing precise information about changed state improved recovery in 38 of 48 cases.
- Adding richer undo operations improved recovery in 142 of 143 cases.
- With the extended recovery language, 180 of 197 defective mutations were recovered; admissible improvements rose from 281 to 461 of 600 tasks.
EvoUndo admission process10:18
- Before a mutation, capture a witness of the original state and define an effect contract and recovery plan.
- Apply the mutation and run its recovery plan in a testing environment.
- Have an independent verifier check whether the original state was restored.
- Accept the change only if it meets capability criteria and passes the recovery check; otherwise repair or reject it.
Example: restoring a timeout12:09
- Read and save the original timeout value of 30 before changing it to 60.
- If recovery is needed, restore the saved value.
- An independent verifier checks that the value matches the original state.
Choose the response to the failure13:18
- If nothing changed, retry safely.
- If the original state is known, restore it; if an external action already committed, use a compensating action where appropriate.
- If safe recovery cannot be verified, stop and escalate.
Step by step
- Define the mutation's effect and recovery plan.
- Capture a witness of the original state before changing it.
- Apply the mutation in a testing environment.
- Run the recovery plan and independently verify restoration.
- Accept the mutation only if it meets capability criteria and passes recovery checks; otherwise repair or reject it.
- For runtime failures, choose among retry, restore, compensation, or stop-and-escalate based on what happened.
Tools & resources mentioned
- Undo Bench — benchmark for evaluating agent task recovery under injected failures.
- EvoUndo — proposed process for capturing state, testing recovery, verifying restoration, and deciding whether to admit agent changes.
- GitHub repository — mentioned as a place to explore EvoUndo; no URL is stated in the transcript.
Notes are AI-generated from the video's transcript and may miss nuance.