# Agents - Skill Authoring and Evaluation A skill body loads into the context of every session that can invoke it, so keep the body lean and move depth into a references directory. Evaluations of agent work need objective metrics, because headline numbers and usage counts mislead systematically. ## Key points ### Structure and discovery - Keep the SKILL.md body lean and move per-skill domain knowledge into a `references/` subdirectory. The body loads into every session's context, so a reader must scan it in seconds. - A skill needs a directory layout (`skills/<name>/SKILL.md`). A flat file (`skills/<name>.md`) is not discovered. A symlink that points to a flat file is not discovered either. - A SKILL.md file requires an `## Instructions` section for the skills linter to pass. An `## Checklist` heading is not a substitute. ### Linter rules and triggers - The skills linter bans generic verb-plus-possessive phrases such as `review my cfp` as invocation triggers. Domain-noun-first forms such as `kubecon proposal review` pass cleanly. - A linter that globs only one plugin tree exempts skills elsewhere. Keep trigger phrases clean anyway, so the skill stays portable when another harness promotes it. - Name a new skill so it does not collide with an existing aggregator skill in the same tree. ### Flags and execution context - Reserve `disable-model-invocation: true` for skills that are destructive on accidental invocation, for rarely-used utilities, and for unavoidably ambiguous triggers. Applied too broadly, the flag breaks auto-routing for skills that benefit from natural-language invocation. - Batch-check the flag across a plugin directory: `yq '.["disable-model-invocation"]' SKILL.md`. - `context: fork` in skill frontmatter gives the skill a copy of the conversation without polluting the main thread. The SKILL.md becomes the subagent system prompt and only a summary flows back. Use it for review and analysis skills. - Splitting a monolithic skill duplicates safety-critical text. A confirmation gate copied into multiple skills can drift apart. Keep one canonical gate file, copy it byte-identically into every gate-bearing skill, and enforce the equality with a check. ### Evaluation of agents and skills - Use binary evals only. Reject subjective 1-5 scales. Run stochastic hill-climbing on the skill or prompt content. Continuous human feedback lets the prompt drift. - State the real hypothesis of an A/B eval first. When one variant inlines what a large document already states, the hypothesis is retrieval cost, not new information. Score objective convention compliance. Scoring `the agent did not open the document` penalizes a variant for following its own instructions. - Dry-run every make target an instructions file names before you ship it. Confirm that the targets, child files, and scripts exist: `make -n <target>`. - A persona-based reviewer needs a when-to-stay-silent discipline as much as finding generation. Approval of a pure deletion pull request with zero inline comments is calibrated. Findings on a clean simplification are miscalibrated. ### Numbers that mislead - Deletion percentage is not a quality metric. One headline reduction of 70 percent contained only 8.5 percent exact duplication. An 84 percent reduction of an instructions file shrank to about 45 percent once upstream and shared content counted toward the working set. Unique rules were lost and cost lines to restore. - Benchmark success rates flatter a model that absorbs errors. A nightly benchmark run swallowed upstream rate-limit errors from an un-staggered fan-out and still scored as a success, so the failure rate understated the real breakage. - Usage counts say nothing about fit. Re-inspection by contents cut 30 removal-candidate plugins to 6. Grep hits were substring matches. Two plugins shipped byte-identical skill sets. Run `diff` on two plugins' skill directories before you disable either. ## How to apply 1. Create the layout: `skills/<name>/SKILL.md` with an `## Instructions` section, a lean body, and a `references/` directory for depth. 2. Write domain-noun-first trigger phrases and run the skills linter. 3. Set `disable-model-invocation: true` only for destructive, rare, or ambiguous skills. Check the whole plugin tree with `yq '.["disable-model-invocation"]' SKILL.md`. 4. Set `context: fork` on review and analysis skills, so heavy output stays out of the main thread. 5. Evaluate with binary metrics and objective convention compliance. Dry-run every make target that an instructions file names. ## Observations - [structure] a lean skill body plus a references directory keeps per-skill domain knowledge out of ambient context #skills (source: agent sessions, 2026-06) - [tooling] a symlink that points to a flat skill file is not discovered, only the `skills/<name>/SKILL.md` layout is #skills (source: agent sessions, 2026-03) - [practice] reserve `disable-model-invocation` for destructive, rare, or ambiguous skills, or auto-routing breaks #skills (source: agent sessions, 2026-04) - [practice] binary evals plus stochastic hill-climbing prevent the prompt drift of continuous human feedback #evaluation (source: agent sessions, 2026-03) - [failure-mode] headline deletion percentages overstated the win and hid lost unique rules #metrics (source: agent sessions, 2026-09) - [failure-mode] usage counts proved nothing about fit: re-inspection by contents cut 30 candidates to 6 #metrics (source: agent sessions, 2026-08) ## Relations - relates_to [[resources/documentation/Agents/Agents - Context and Cost Economics]] (a lean skill body keeps the per-session context budget small) - relates_to [[resources/documentation/Agents/Claude Code - Plugins, Settings, and Scheduled Tasks]] (the plugin tree that holds the skills) - relates_to [[resources/documentation/Agents/Agents - Guards, Hooks, and Security Boundaries]] (a confirmation gate copied byte-identically across gate-bearing skills) - relates_to [[resources/documentation/Agents/Agents - One Memory, Many Harnesses]] (trigger-phrase hygiene keeps a skill portable across harnesses) ## Sources - Personal agent-session notes, February 2026 to October 2026