Most skills hand back text or a file — a mistake there is a bad paragraph. A skill that writes to a live database or content store is different: a mistake is duplicated, corrupted, or lost data. The build discipline has to change with it.
--confirm gate for live, content-addressed idempotency, verify-before-write, per-item commits, and a read-back check), then smoke-test once against a disposable test target and prove idempotency by re-running rather than benchmarking. The code earns trust from real use plus that one clean re-run, not from a score.A store-mutating skill is any skill whose success is a side effect on persistent shared state, not a returned artifact. “Vault these images with metadata,” “register these records,” “promote this migration” — the output isn’t text you can eyeball and discard; it’s rows and files that other systems and people will read afterwards. That single fact reshapes how you build and how you test it.
Table of Contents
Why the usual skill eval loop is the wrong tool here
The normal way to harden a skill is a loop: run it against a handful of realistic prompts, once with the skill and once without (the baseline), grade the outputs, and iterate. It’s a good loop — for skills that produce text or files, where each run is cheap and disposable.
Point that loop at a store-writing skill and it turns destructive. Every “with-skill” eval run performs real writes; every “without-skill” baseline run also tries to, flailing without the skill’s guardrails. Ten eval prompts across a few iterations is dozens of write passes into a store you now have to clean up — and the baseline runs are actively the unsafe case you built the skill to prevent. You’d be benchmarking by making the exact mess the skill exists to avoid.
A benchmark for a store-writing skill is just a controlled way of dirtying the store.
So skip it. Not because testing doesn’t matter — because that test is the wrong shape for this risk.
Smoke-test instead of benchmark
Replace the benchmark with two cheap, honest checks:
- One smoke run against a disposable test target. Point the skill at a throwaway test store (never production), run it on a small real input, and verify the result end to end — rows written, files present, metadata attached, read back correctly. One pass, real data, fully inspected.
- A re-run to prove idempotency. Run the same input again. A correctly built store-writing skill writes nothing new the second time — same identities, no duplicates. If the second run is a clean no-op, the skill is safe to re-invoke, which is most of what “safe” means for a mutating tool.
That’s it. There’s no baseline-vs-skill score because the skill’s value isn’t a quality delta a grader can read — it’s correct, safe writes, and correctness is verified by inspection and idempotency, not by comparison.
The guardrails belong in the script, not the prose
A store-writing skill’s safety must live in its bundled script — executed every invocation, not restated in the SKILL.md body where the model might paraphrase or skip it (see the script tier in Anatomy of a Claude Code Skill). Five rails earn their place:
- A target is always required — no silent default to production. The script demands an explicit
--testor--live, and a live write additionally requires a--confirm=<name>matching the target. A forgotten flag must fail, never quietly hit prod. This one rail prevents the worst mistake outright. - Idempotent on content. Derive each record’s identity from its content (a hash) or upsert on a natural key, so re-running never duplicates. Idempotency is what makes the re-run test meaningful and what makes a resumed batch safe.
- Verify before you write. Validate each item and abort loud on a bad one rather than writing garbage. For images, that means rendering the file and checking it isn’t blank or broken before it enters the store — a thing that passes a text check can still be wrong.
- Commit per item, resumably. Write and commit one record at a time so a failure mid-batch leaves a consistent partial state you can simply re-run into, not a half-applied mess.
- Read back after writing. Confirm a sample landed as intended — the file exists on disk, the record resolves by its key. A write that returns success but stored nothing is the quiet failure this catches.
In practice, building a skill that batch-vaults images into a content store, all five of those lived in one script the SKILL.md called; the body just explained the model and pointed at a field reference. We proved it with a single smoke run against the test store and one idempotent re-run — no benchmark — and then trusted it because it had also done the real work correctly. The guardrails, being code, apply to every future invocation for free.
Put the safety in the code path that always runs, not the prose the model might skim.
Explicit –test or –live; live also needs –confirm=<name>. No default to prod – a forgotten flag FAILS, never hits production.
Identity from a content hash or an upsert on a natural key, so re-running never duplicates. This is what makes re-run safe.
Validate each item; render an image to catch a blank/broken one. Abort LOUD on a bad item rather than storing garbage.
One record at a time, so a mid-batch failure leaves a consistent partial state you simply re-run into.
Confirm a sample landed by its key. Then prove it: one smoke run on a test target + a re-run that must be a clean no-op.
A benchmark for a store-writing skill is just a controlled way of dirtying the store.
The short version
If a skill writes to a real store, don’t benchmark it — benchmarking means dozens of junk writes, and the baseline runs are the unsafe case itself. Build the safety into the bundled script: a required target with a --confirm gate for live, content-addressed idempotency, verify-before-write, per-item resumable commits, and a read-back check. Then smoke-test once against a disposable test target and re-run the same input to prove it’s a clean no-op. Correctness comes from inspection and idempotency, not a score — and because the rails are code, every later invocation inherits them.
Frequently Asked Questions
Why not just benchmark a store-writing skill like any other?
Because each eval run performs real writes, and the baseline (no-skill) runs try to write without the guardrails — so the benchmark itself dirties the store and exercises the unsafe path. The risk is the writes, and a benchmark multiplies them.
How do I test it, then?
One smoke run against a disposable test target, inspected end to end, plus a re-run of the same input to confirm it writes nothing new. Idempotency + read-back is the safety proof for a mutating tool.
What stops it from writing to production by accident?
A required target flag with no default, and a --confirm=<name> gate on live writes. A forgotten flag fails loudly instead of silently hitting prod — the single most important rail.
Where should the safety checks live — in SKILL.md or a script?
In the bundled script, which executes every time. Instructions in the body can be paraphrased or skipped by the model; code in the script path can’t. The body should point to the script, not restate its guarantees.
Does the store-writing skill still get committed and tracked like other skills?
Yes — same rule as any skill: commit it into the repo whose domain it serves rather than leaving it personal (see Where Claude Code Skills Live).
