AIE CODE: Google DeepMind, Anthropic, SpaceXSI, TypeSafe, Lovable, Ramp, etc (System One theme)
Make agents more dependable by improving their environment: use deterministic checks, strong verification, staged autonomy, and safe recovery.
- Turn recurring agent mistakes into code-level constraints—lint rules, types, architecture, and tests—instead of relying only on prompt rules.
- Build a verification loop around agent work: run it, inspect evidence, gate merges by risk, and check behavior after deployment.
- Increase autonomy gradually, with clear human approval points, limits on task volume, and a tested way to undo or contain changes.
The talks offer useful frameworks and examples, but the notes capture most of the practical takeaways. · Best for: Engineers and technical leads building coding-agent platforms, workflows, or software factories.
A conference compilation on building and operating coding agents. Across talks, the recurring themes are inference speed, simpler agent harnesses, reliable verification, shared operational context, safe autonomy, and recoverability.
Faster inference and system-one models11:26
- Faster inference can keep users in flow and enable online agent use cases; as inference speeds up, tool calls can become the new bottleneck.
- Gimlet's approach splits model workloads across accelerator types to improve the trade-off between token speed and throughput.
- Cloudflare describes pairing fast decision and extraction models with larger language models: use small models to route, extract, or prune context, and reserve LLM calls for harder reasoning.
- In the CI-triage example, the system-one workflow reached a similar answer with fewer LLM calls and tokens.
Build trust into code and agent operations1:01
- SpaceX Super Intelligence's approach is to make the codebase a reliable input: encode conventions in APIs, architecture, lint rules, types, and tests so agents follow good patterns by default.
- Verify behavior with runtime tools and artifacts such as screenshots or videos; the talk also describes experimenting with formal and property-based verification.
- Harvey's agent platform runs existing agents in a shared cloud harness with secrets, sandboxes, follow-up loops, and a common review process.
- Limit automated task volume and add backpressure: a repeated alert should join an existing investigation rather than start duplicate work.
Lean harnesses, branchable environments, and safe factories2:21
- The Google DeepMind talk argues that capable models may need fewer specialized tools; test a Bash-only harness and compare cost and token use as well as task success.
- Databricks describes branching agent state—including databases, memory, and files—so agents can investigate multiple hypotheses in isolated environments.
- Sonar's safe-factory framework adds checks at PR review, after deployment, during ongoing maintenance, and through a shared knowledge loop.
- Factory recommends staged autonomy: start with human checkpoints, then raise autonomy as quality and validation evidence justify it.
Self-improvement and undo3:59
- The Klein talk describes running an agent against evaluations, grouping failures, changing the harness, and rerunning tests to improve performance.
- Undo research finds that completing a task does not guarantee recovery from failures, especially when external side effects or runtime configuration changes are involved.
- Capture the pre-change state, apply the mutation, attempt recovery, and have an independent verifier check that the original state was restored.
- Use safe retries before a mutation, restore known state when possible, compensate for committed external actions, and stop and escalate when recovery cannot be verified.
Automating test generation and assessing agent skills4:42
- Duolingo's test pipeline selects suitable files, generates tests, checks CI and test quality, routes PRs for human review, and feeds reviewer feedback into later runs.
- The team prioritized testable MVVM files, avoided UI files for this workflow, capped automated repair attempts, and used Mac runners to validate iOS code locally.
- For risky test PRs, deterministic rules and an LLM reviewer reject changes that alter production code, disable tests, or introduce timing-based waits.
- The Zenity talk warns that a skill's visible instructions may not reveal its full behavior; it describes running skills in a sandbox with a constructed environment and analyzing execution traces.
Reliability, memory, and human intent5:06
- The Jev talk argues that assistance and dependable automation are different objectives; optimize for narrow, repeatable behavior when the system must act without constant human intervention.
- It calls for testing software ideas that place intelligence in the product's inner loop, rather than treating AI only as a coding assistant.
- The available transcript ends during this talk, so its remaining claims and examples are not covered here.
Step by step
- Identify one recurring agent task or failure that has a clear success condition.
- Improve the deterministic environment first: clean up misleading code patterns and add enforceable rules, types, or tests.
- Give the agent the minimum useful tools and context; compare harnesses on quality, latency, and token or cost use.
- Run the agent in a sandbox or branchable copy where it can test changes without affecting shared state.
- Collect evidence through CI, runtime checks, review artifacts, or an independent verifier.
- Start with human approval, then expand autonomy only for actions with low mistake costs and reliable detection or recovery.
- Log failures and corrections, feed recurring lessons into code or shared skills, and cap retries or task volume.
Tools & resources mentioned
- Token Mint Factory — game used to illustrate the trade-off between token speed and throughput.
- Cloudflare Workers AI and system-one models — described for fast decisions, extraction, routing, and reducing LLM workload.
- Dune — constrained client framework and lint-rule approach discussed in the SpaceX Super Intelligence talk.
- TLA+, Lean, and Bombadil — tools named for model checking, algorithm proofs, and UI property-based testing.
- Chrome DevTools Protocol — runtime debugging interface used by agents.
- Gemini Decision API and Gemini 4 Carbon — Google DeepMind models and API discussed for decisions and coding-agent experiments.
- Neon — branchable database foundation described for isolated agent work.
- Temporal, Amazon S3, GitHub Actions, and Datadog — Duolingo pipeline orchestration, shared state, repository automation, and observability.
- Undo Bench and Evo Undo — research projects for measuring agent recovery and verifying restoration after changes.
- Skills and playbooks — shared instructions and team knowledge used by agent workflows; the talks emphasize testing and maintaining them.
Notes are AI-generated from the video's transcript and may miss nuance.