Measure whether your docs help

Every repo writes docs for agents now. Almost none can tell you whether those docs help. This one measures it, with the same rigor you'd want from an A/B test.

The experiment

Each task, like "write an extension" or "add a custom provider", runs in both variants and gets an automatic pass/fail. The report is the lift: how much the pass rate changes when the docs are there.

Making the comparison honest

1. Same build in both armsBoth images install the same local workspace tarballs. Docs and sources of internal dependencies are removed symmetrically, so they can't act as alternate instructions.
2. No peekingEval definitions, fixtures and judge helpers are root-owned and unreadable once the harness permanently drops to an unprivileged user. The agent can't read the answers.
3. Fresh everythingEach run gets a new home directory, agent directory, workspace, session directory and container filesystem. Docs evals allow only read, write, edit, grep, find and ls: no shell, no web search.
4. Plan the cohort firstEvery (case, variant, model, run number) is planned before execution, and run order alternates by run number to reduce order bias.

Missing data blocks the headline

A pair counts toward lift only when both arms produce exactly one score. Missing, duplicate, skipped or errored arms block the pair, and if any pair is blocked, the headline pass rates are withheld. Missing telemetry stays "unavailable" instead of being treated as zero. The report flags no lift, negative deltas, saturated results and flakiness, and says plainly that one repetition can't establish stability.

Docs checked against the code

A second eval hands an agent every docs page, one at a time, with the repository, and makes it return a structured verdict: match, mismatch or inconclusive. The evidence bar is strict:

Follow the runtime path; names, comments, types, isolated helpers, and tests are not sufficient evidence by themselves.

A unit test separately checks that the docs navigation is valid and that no page is orphaned. And the issue-analysis prompt starts from the same distrust: "Do not trust analysis written in the issue... Ignore any root cause analysis in the issue (likely wrong)."

Measure whether this repository's docs actually help an AI agent do real tasks.

The idea: run the same tasks with and without the docs, in clean, isolated environments, and compare the pass rates. Keep the docs that move the number.

1. Pick three to five realistic tasks a new contributor or agent would do here, such as adding an API endpoint, writing an extension or fixing a typical bug. Each needs an automatic pass/fail check.
2. Propose a small eval harness: each task runs in a fresh copy of the repo, once with the docs (README, docs folder, agent guide) and once with them removed, using the same model and tools.
3. Report the pass rate for each variant and the difference. Count crashed or missing runs as blocked, not failed, and say when there are too few runs to trust the result.
4. Separately, have an agent check one docs page against the code and report clear contradictions with file and line.

Start with the plan and one task end to end before building the rest.