Your Spec Has Bugs. Catch Them Before Your Agent Codes. - Rajath Mahendrappa Kotyal, AWS
Resolve conflicting, vague, or missing specification rules before coding by translating them into logic, checking them with a solver, and asking a human to decide unresolved cases.
- Check specifications for conflicts between rules, ambiguous wording within a rule, and gaps where no rule covers a case.
- Translate unclear rules into SMT formulas and use a satisfiability checker to flag inconsistencies.
- Compare multiple LLM interpretations by logical equivalence; ask the user to clarify or abstain from implementing unresolved rules.
The workflow and examples are explained, but there is no detailed tool walkthrough. · Best for: Teams using coding agents to implement business rules or written specifications.
The talk describes a pipeline for finding specification defects before a coding agent writes code. It uses LLMs to refine wording, formal logic and a solver to detect issues, and human judgment to resolve cases requiring domain knowledge.
Why specifications need review0:00
- Keto Specification Analyzer applies formal methods and automated reasoning before a specification reaches a coding agent.
- Example conflict: one rule says unavailable inventory creates a back order; another forbids back orders.
- The speaker cites research reporting accuracy drops with unclear prompts, but the talk does not detail the research sources here.
Detecting ambiguity and contradictions2:05
- The system checks wording within individual sentences and relationships across sentences for conflicts and gaps.
- It first asks an LLM to refine wording; if that does not resolve the issue, it formalizes the English as SMT formulas and checks them with a satisfiability solver.
- Detected ambiguity or conflict is sent back for review and another iteration.
Comparing possible interpretations4:08
- The system asks the LLM to translate a rule multiple times—seven or ten examples are mentioned.
- It clusters translations by logical equivalence to identify distinct interpretations and their assumptions.
- It then asks the user to clarify, or abstains from implementing the rule.
Human decisions and limits5:05
- For a vague instruction such as “Remove the record,” the user must choose what removal means, such as deleting the record or hiding it from users.
- A conflict means rules contradict; a gap means the specification does not cover a case, such as an idle order.
- The user remains the authority on correctness because the LLM lacks implicit domain knowledge; the tool assists rather than guarantees correctness.
Takeaways6:57
- Review the specification before passing it to an agent.
- Cluster interpretations by logical equivalence; the LLM translates, the solver checks, and the human decides.
- If there is no answer, use LLM judges or abstain from implementing that piece.
Step by step
- Have an LLM try to refine unclear specification wording.
- If the issue remains, translate the rules into SMT formulas and check them with a satisfiability solver.
- For flagged rules, generate multiple LLM translations and cluster them by logical equivalence.
- Ask the user to resolve distinct interpretations or conflicts; abstain from implementing unresolved rules.
Tools & resources mentioned
- Keto Specification Analyzer — project presented for analyzing specifications before coding agents use them.
- SMT formulas — formal representation of specification rules for checking.
- SAT solver / satisfiability checker — evaluates whether formalized expressions are valid or conflicting.
- Team blog and website — mentioned as further reading; no URLs are given in the transcript.
Notes are AI-generated from the video's transcript and may miss nuance.