Pick Problems That Survive When AI Solves Coding - Ara Khan, Cline
Choose work that benefits from AI progress and helps improve the AI pipeline, rather than tasks a model release could eliminate.
- Look for gaps in the full AI pipeline—evaluation, harnesses, training, inference, hardware, or real-world interaction.
- Use verifiable tasks and feedback to find failure patterns, improve the system, and test changes across many problems.
- Favor ambitious work that benefits from scale but cannot be solved by scale alone or wiped out by the next model release.
The notes capture the main ideas; the evaluation-loop example adds some implementation detail. · Best for: Software engineers and builders reconsidering what to work on as coding agents improve.
Ara Khan describes recursive self-improvement through a coding-agent evaluation loop, then argues that engineers should choose problems that both benefit from AI progress and remain valuable as models improve.
Why recursive self-improvement matters0:17
- The talk aims to explain frontier-lab direction, career choices, and the risk of being left behind as AI advances.
- The speaker recommends reconsidering which problems are worth pursuing as coding agents make routine work easier.
How coding agents can improve their harness2:55
- Cline’s harness scored below the model release paper’s reported benchmark result, prompting efforts to improve its performance.
- A meta-agent can run evaluations, inspect failure patterns, change the harness, and repeat the tests.
- The speaker describes a 17-hour run that improved a baseline score of 27%; it identified issues such as loops, long-running background processes, and file indexing.
- Changes should improve performance across a broad range of tasks, not merely exploit a particular evaluation.
Evaluation and the broader improvement loop8:09
- Agent evaluation can involve multi-step tasks, isolated sandboxes, independent graders, and measures such as success, tokens, latency, and tool calls.
- The speaker connects improvement loops to training, inference, and physical systems, where outcomes can be tested and used to guide changes.
- Human corrections can reveal capability gaps that future training or system improvements may address.
Choose problems that persist as models improve21:33
- The speaker argues for work closer to the AI improvement pipeline, including inference, post-training, hardware, networking, and real-world interaction.
- Prefer ambitious, targeted projects that benefit from AI progress but are not solved by scale alone.
- If a model can routinely complete your work in one shot, reassess whether your effort is contributing distinctive value.
Step by step
- Run an evaluation suite on the agent in isolated environments and collect task results.
- Group failures into patterns, such as looping, slow execution, or tool and indexing problems.
- Have a meta-agent propose harness changes based on those failures.
- Rerun the evaluation and check whether the changes improve results across tasks without exploiting the benchmark.
- Choose future work by asking whether it benefits from AI progress, contributes to the improvement pipeline, and survives the next model release.
Tools & resources mentioned
- Cline — coding-agent harness discussed in the evaluation example.
- Terminal Bench 2.1 — terminal-agent benchmark used to evaluate coding-agent performance.
- GPT-5.6 — model described as analyzing evaluation failures and modifying the harness.
- OpenRouter — API access mentioned for experiments with a hardware device.
Notes are AI-generated from the video's transcript and may miss nuance.