Creator Digest
AAI Engineer· 1M subscribers

Your Agent Doesn't Need an LLM for Every Decision - Raymond Weitekamp, OpenProse

52 min videoNov 10, 2026The notes cover it

Use fast decision models to handle small, bounded choices, and call an LLM only when a task needs generation or open-ended reasoning.

  • Keep task context outside the model and pass focused slices to recursive decision calls.
  • Compose small decisions into search or control loops; set a maximum recursion depth.
  • Use an arbiter to resolve competing actions when multiple decision models run in parallel.

The architecture and examples are covered in the notes; watch only for the demos. · Best for: AI engineers building agents, computer-use systems, or low-latency interactive interfaces.

Raymond Weitekamp proposes recursive decision models (RDMs): systems where fast classifiers make bounded choices, can call smaller decision processes recursively, and use an LLM only when generation is needed. He demonstrates the approach with search, games, smart-home control, and real-time audio, then describes OSO, a voice-first computer-use project.

Decision models and recursive language models0:00

  • Decision models rank or select among provided options; unlike LLMs, they do not generate open-ended text.
  • RDMs adapt the recursive structure of recursive language models to decision models.
  • The proposed design makes a decision model the root and lets it call other decisions or, when needed, an LLM or agent.

RDM rules and decisions all the way down13:05

  • Keep context outside the model so code can pass relevant slices.
  • Allow decisions to call other decisions, with a maximum recursion depth.
  • Require the design to work without a generative model, while still supporting LLMs and agents.
  • Examples include composing decisions to control a game and navigate Factorio without an LLM.

Hierarchical search and the limits of fixed decisions17:16

  • Use decisions at successive levels of a category hierarchy to narrow a search instead of evaluating every item at once.
  • The same zoom-in pattern can be applied to image regions.
  • Decision-only systems need their possible actions or paths to be enumerated or programmatically defined.

Combining decisions with generation25:25

  • Use decisions for bounded choices and an LLM when a task requires open-ended generation.
  • The smart-home example uses a shared map of actions: the decision model selects an action, while code executes it.
  • The LLM can add or edit map entries based on user requests, such as learning a shortcut for a routine.

Parallel decisions and arbitration31:39

  • Fast decision models can run speculatively in parallel.
  • An arbiter gates which proposed action can change system state, avoiding conflicting writes.

Low-latency interaction and OSO35:07

  • The speaker describes RDM implementations in a DSPy module and a standalone Rust package.
  • In the Earshot demo, a small model learns to classify short audio slices while an RDM decides when to trust it.
  • OSO is described as a planned voice-first computer-use agent that learns user preferences and delegates decisions to smaller models as trust develops.

Step by step

  1. 1. Define the task state and keep it outside the decision model.
  2. 2. Enumerate available choices or create a hierarchy of choices that code can traverse.
  3. 3. Give each decision call only the relevant context slice and define what choices it can return.
  4. 4. Allow recursive calls where useful, and enforce a maximum recursion depth.
  5. 5. Add an LLM or agent for parts requiring open-ended generation; use a shared action map to connect generated behavior to executable actions.
  6. 6. If decisions run in parallel, use an arbiter to select which action is allowed to update state.

Tools & resources mentioned

  • TypeSafe Jev — decision model used in the examples.
  • Recursive coding agents talk — earlier talk the speaker recommends for more on recursive language models.
  • DSPy module — one implementation path for the RDM prototype.
  • RDM Rust package — standalone implementation for portable systems.
  • OSO (oso.computer) — planned voice-first computer-use project; described as not yet ready.
  • Hunch with Gemma 2 and WebGPU — on-device doodle-classification project mentioned as a related example.
  • Wekinator, ChucK, and Chai — tools or frameworks referenced in the speaker’s real-time music and audio-machine-learning work.
Watch on YouTube

Notes are AI-generated from the video's transcript and may miss nuance.