Your SKILL.md Is 70% Stuff the Model Already Knows - Nikolay Petrov, Intel
Measure which skill details a model already knows, then remove redundant content only if task evaluations stay intact.
- Compare model answers with and without a skill on representative tasks to verify it improves results.
- Test skill knowledge items in both conditions; flag items that add knowledge, do nothing, or degrade answers.
- Remove redundant details only when task evaluations confirm performance is unchanged.
The evaluation method and decision framework are explained clearly in the notes. · Best for: AI engineers building, evaluating, or optimizing agent skills.
The talk presents a way to distinguish new knowledge in a skill from information a model already knows. Test skill content and task performance with and without the skill, then use the results to guide safe reductions.
Evaluate skills on tasks0:00
- Skills can add knowledge or provide a repeatable process; this talk focuses on skills that add knowledge.
- Define tasks, run them with and without the skill, and compare success along with costs such as tokens and time.
- The example improves from three of five tasks solved without a skill to all five with it.
Break skills into testable knowledge items3:16
- Rather than edit skill text by guesswork, split it into items and define an expected question and answer for each.
- Test a set of roughly 25–30 items per skill in both conditions.
- Use a judge or panel, ideally from a different model family, and ensure questions do not reveal their answers.
Interpret the results6:02
- If the model answers correctly both with and without the skill, that item adds no new knowledge.
- If adding the skill makes answers worse, revise or remove the harmful content.
- If only the skill-enabled model answers correctly, the skill adds knowledge; if neither succeeds, the item may be unclear or too difficult.
Check general understanding versus exact details8:56
- Use lenient checks for general concepts and strict checks for exact parameters, values, or steps.
- The talk reports a large gap for Opus: 70% understanding on general checks versus 22% on specific details.
- Skills can still be useful for supplying exact details that models do not reliably recall.
Use findings to optimize skills10:30
- Knowledge overlap varies by model and skill, so test across a broader set of models where possible.
- Novelty results are an initial optimization signal, not a substitute for task evaluations.
- Try removing already-known content, then rerun task evaluations; keep reductions only if results remain stable.
Evaluation framework11:54
- Docume is described as a framework for comparing agent-related context, including skills, agents.md, and MCP.
- The framework includes novelty analysis as well as evaluations of context impact, routing, descriptions, and naming.
Step by step
- Define representative tasks the agent should solve and run them with and without the skill.
- Break the skill into knowledge items; write a question and expected answer for each item.
- Ask the model each question in both conditions and score answers with a judge or panel; avoid questions that leak answers.
- Classify items as already known, newly learned, harmful, or not learned in either condition.
- Use the results to revise or remove redundant or harmful content, then rerun task evaluations before keeping changes.
Tools & resources mentioned
- Docume — framework mentioned for evaluating skills and other agent context.
- Sonnet and Haiku — models named in the reported evaluation cost example; Sonnet was used as a judge.
- Opus — model used in the comparison of general versus exact-detail understanding.
Notes are AI-generated from the video's transcript and may miss nuance.