AIE CODE: OpenAI, ArtCraft, Cognition, Exa, Anthropic, Amp, et al
Use AI to expand what you build, but keep humans accountable for quality, measure agent workflows, and improve them from real usage.
- Review agent output actively; define quality criteria and iterate instead of accepting one-shot results.
- Track agent sessions and repeated workarounds; turn common output filters into CLI options and test changes against recorded calls.
- Automate checks for verifiable facts, reserve judgment calls for agents, and escalate sensitive or uncertain decisions to people.
Many talks offer useful examples, but the notes capture their main practices and takeaways. · Best for: Engineering leaders and developers building, evaluating, or governing AI coding-agent workflows.
A conference program on ambitious AI-assisted development, software factories, agent-friendly tools, and quality controls. Speakers emphasize that better models increase the value of human judgment, clear evaluation, reliable tests, and careful workflow design.
Ambition, ownership, and taste18:49
- OpenAI speaker Jason Liu argues that AI lowers execution barriers, so developers should take ownership of the work they ship rather than passively accept generated output.
- Build the ability to describe what good looks like: study products, design, games, film, and other fields, then apply that judgment when iterating.
- Use AI to explore interests and disciplines you previously set aside; ambition is about pursuing bigger problems, not simply using more tokens.
Build a workshop before a software factory40:08
- The Pi coding-agent speaker says model improvements have made some older harness tricks unnecessary, while agents run longer and need less correction.
- Starting projects is getting easier, but maintaining them is not; the speaker's project count rose while the number maintained regularly stayed roughly flat.
- Preserve engineering judgment, shared standards, and human collaboration. Use summaries during agent work to understand changes without rereading every line of code.
Factories, open tools, and agent usability1:00:46
- Warp CEO Zach Lloyd defines a software factory as a code-defined, measurable, closed-loop system for improving cloud coding agents.
- The proposed stack includes factory configuration as code, stored agent conversations and artifacts, sandboxed compute, support for multiple models and harnesses, orchestration, and API/MCP integrations.
- A separate CLI talk recommends mining transcripts for repeated pipes and filters, then adding selection flags, compact output, pagination, and clear failure summaries.
- Artcraft's speaker describes building open-source creative applications with agents and Rust, while emphasizing community contributions and human direction.
Agent traces, tuning, and scalable engineering1:59:06
- Brian Douglas uses session traces and human pushback as signals for what agents do not know; he describes using those examples to create skills and experiment with reviews and model tuning.
- A VLM optimization demo uses Optuna with Bayesian optimization to test serving configurations and produce an HTML report and optimized YAML recipe.
- Cognition describes moving engineers into smaller, more vertically responsible teams, running agents in reproducible local Docker Compose environments, and assigning long-lived goals rather than isolated coding tasks.
- Cognition's examples use agents to improve latency and GPU throughput, and to maintain Android parity with an iOS app through ongoing tests and user feedback.
Search and memory for longer tasks2:59:45
- Exa argues that model memory is incomplete and becomes outdated; its Atlas benchmark tests whether agents can find and combine correct information across sources.
- The talk contrasts keyword search with semantic retrieval, partner and knowledge-graph data, and concise extracted answers intended to reduce irrelevant context.
- Anthropic's talk presents multi-agent coordination using isolated sandboxes, Git, and a shared database, plus memory that agents can write, review, and reorganize.
- The speaker describes 'dreaming' as processing transcripts and notes into more coherent representations, such as wikis, and correcting or filling gaps in existing memory.
Factories, reviews, and safe changes3:23:23
- Amazon's Kiro Crew account describes progressing from individual sessions to parallel sessions, scheduled work, pipelines, and an orchestrator that can unblock agents.
- Its review process uses multiple reviewers, separates blocking from non-blocking findings, tests changes, and checks a release build after merge.
- Robinhood's data-pipeline reviewer combines deterministic scripts for authoritative facts with an agent for code-context risks and people for sensitive or uncertain decisions.
- The Robinhood example shows that green CI and registered lineage can miss an unregistered downstream consumer; the agent found the code path and escalated it for human confirmation.
Large-scale changes and final discussion5:20:06
- The Bun compiler talk says large agent-written projects are constrained by verification: reuse trusted pre-existing test suites where possible, compare performance, and fuzz behavior.
- The presenter rewrote an ahead-of-time compiler several times, balancing code quality, runtime memory and startup improvements, and compilation time before deciding the latest version was ready to ship.
- The Syntax discussion covers explicit state machines, agent-oriented languages and frameworks, Rust and native UI, and the continuing need for engineering judgment and product care.
- Across the talks, high output is not itself the goal: maintainability, reliable checks, clear ownership, and attention to users remain important.
Step by step
- Define the agent workflow as code: triggers, tools, policies, environments, and the stages work passes through.
- Run agents in reproducible sandboxes and record their conversations, artifacts, costs, outcomes, and human interventions.
- Create evaluations and benchmarks from real tasks; replay them when changing models, harnesses, skills, or factory configuration.
- Inspect transcripts for repeated workarounds. Add explicit CLI options for common filters or slices, make failures clear, and test that relevant output is not lost.
- Use deterministic checks for facts available from authoritative systems; use agents for contextual review and escalate sensitive, high-risk, or uncertain decisions.
- Build tests and release checks into the workflow, then use measured failures and human feedback to update rules and agent instructions.
Tools & resources mentioned
- Pi coding agent — coding-agent harness discussed in the workshop-versus-factory talk.
- Warp factories — demo of code-defined agent workflows, monitoring, evaluations, and benchmarking.
- Artcraft — open-source creative applications and infrastructure described by its founder.
- Tapes and Paper Console — session-trace capture and review tools discussed in the pushback talk.
- TwinReviewer — local PR-review experiment based on labeled traces; the speaker said it was not open source.
- Optuna — optimization search tool used with Bayesian optimization for VLM serving recipes.
- Atlas — Exa's benchmark for testing agent knowledge retrieval across conditions and sources.
- Kiro Crew — open-source system for running multiple Kiro agents; the speaker also mentioned a workshop and free trial.
- Lean and Prove2Me — formal-proof system and orchestration platform discussed in the multi-agent mathematics example.
- Docker Compose — used to run services and mocked dependencies locally for agent testing.
- Bun test suite, WebKit tests, Test262, and a JavaScript-engine fuzzer — existing tests and fuzzing used to verify the ahead-of-time compiler.
- Rick Rubin, The Creative Act — book recommended in the session on agent traces and learning through practice.
Notes are AI-generated from the video's transcript and may miss nuance.