Engineering Agency out of the Happy Path — Matthew Jewkes, Standard Cybernetics
Automate stable, high-volume, specifiable work—and use AI to route ambiguous, risky, or novel cases to experts.
- Choose processes with clear business value, steady volume, and detail-oriented work; avoid subjective judgment and early-stage discovery.
- Build input filters and escalation paths for ambiguous, unusual, or business-critical cases before expanding automation.
- Create and review about 1,000 evaluation examples with domain experts; improve the system by analyzing failures and testing against historical work.
The notes capture the framework and case study; the talk offers few step-by-step implementation details. · Best for: AI and engineering leaders automating established processes at growth-stage companies.
Matthew Jewkes argues that full automation should handle routine cases without humans in the normal path, while AI screens inputs and routes exceptions to people. A case study shows how this can improve task quality and organize expert attention around unresolved work.
Why full automation matters0:14
- Experts often lose time to multitasking and routine decisions, leaving less attention for high-value work.
- As execution gets cheaper, organizational judgment becomes more important.
- Full automation removes people from the normal workflow, while retaining them for exceptions.
Use AI as an escape hatch2:09
- Agents can be difficult to maintain and may generate extra decisions for leaders.
- Automations are legible: they take inputs, produce outputs, and encapsulate work.
- Use AI to screen for out-of-scope, adversarial, or business-critical cases and escalate them.
Case study and evaluation4:24
- The system indexed documents and their relationships, processing tens of thousands of tasks per month.
- The team started with a research-and-assignment agent and reached about 83% performance after a few sprints.
- They compiled 1,000 evaluations and brought experts completed examples to review rather than asking them to start from a blank spreadsheet.
Route errors, ambiguity, and exceptions6:34
- Separate critical errors, cases where experts disagree or the organization has not decided, and unusual cases requiring escalation.
- Screen and categorize uncertain inputs; send critical work through a heavily tested pipeline, then escalate unresolved cases.
- After excluding ambiguous and unusual cases, the team improved accuracy on the critical set from about 95% to 99.95%.
Turn automation output into useful expert attention10:49
- The system's remaining cases give experts organized queues of critical reviews, ambiguous work, and exceptions.
- Reviewing ambiguous cases together can help decide whether to specify, prioritize, or stop doing that work.
- Aggregated exceptions can reveal patterns and help experts identify future business investments.
Where to invest and how to improve14:08
- Full automation can save time, improve outcomes, reduce organizational complexity, and redirect talent to higher-value work.
- Favor established processes with material ROI, written specifications, and detail-oriented tasks; avoid startups in discovery and subjective judgment work.
- Improving from roughly 95% to 99.95% can be costly. Read and write evaluations, build a large evaluation set, and inspect failures.
The value of attention and judgment17:39
- The speaker argues that AI's lasting advantage may come from helping organizations direct collective attention and judgment.
- When work can be specified, delegate it so experts can focus on opportunity and higher-value decisions.
Step by step
- Select an established, high-volume process with clear business value, preferably one with written specifications and detail-oriented work.
- Work with domain experts to assemble and review a large set of evaluation examples; inspect specific model failures.
- Separate routine cases from ambiguous cases, critical cases, and unusual exceptions.
- Use AI to screen incoming work and route each category to its appropriate workflow or human reviewer.
- Tune and test the critical-case pipeline; deploy only after a random sample performs better than historical work.
- Use the accumulated exception and ambiguity queues to guide expert decisions and identify future investments.
Tools & resources mentioned
- GPT-5.5 — model mentioned in an example of differing answers on an ambiguous drug-target question.
- Claude Opus 4.8 — model mentioned in the same ambiguous-case example.
- BERT — model family the speaker's team had previously fine-tuned for NLP pipeline components.
Notes are AI-generated from the video's transcript and may miss nuance.