Measuring a dead-code detector honestly: the metric was the bug.

Direct answer

What is the short answer?

To measure a dead-code detector honestly, score it against per-symbol ground truth, not against a blanket "clean corpus" assumption. Label every exported symbol as verdict_source (genuinely dead), detector_fp (a real false positive), or not_findings (correctly left alone). Then split the detector's output into two populations: oracle-confirmed verdicts and text-only candidates. Report precision on the verdicts only.

On our CD03 unused-export detector, measured in August 2026, this flipped the headline. Precision went from 0.39 (CI [0.24, 0.56], a confirmed floor failure) to 0.989 (CI [0.938, 0.998], a confirmed pass). Zero detector source changes happened between the two runs. The recall cost of the verdict-only contract (0.509) is a property of running on a partially-indexed package, not a detector defect.

The bug

Why does a clean corpus make every hit look like a false positive?

The natural way to measure a false-positive rate is simple. Take a corpus you believe is clean. Run the detector. Count the hits, divide by files. Every hit is an FP because the corpus is clean. This is the v2a methodology we used in the previous measurement.

It has a hidden assumption: the corpus is actually clean at the granularity the detector measures. For CD03 — "this exported symbol has no cross-file references" — the unit is the exported symbol, not the file. A mature library like zod is "clean" in one sense. It has no dead code the maintainers know about. But zod exports hundreds of symbols. Some genuinely have no references within the sampled file slice we gave the detector. The consumer (zod/mini, an external sibling package) lives outside the slice.

Under the naive metric, a correct finding is scored as a false positive. "This symbol is unreferenced in the slice you showed me" is true. But we asserted the corpus was clean, so it counts against the detector. The detector was right. The accounting called it wrong. That is the bug.

The blind truth table (v3)

The v3 fix: a per-symbol source-truth table

The correction replaces the blanket assumption with explicit labels. We built a truth table over the same 100-file clean corpus. It holds 953 labeled exported symbols:

The labeling is deterministic, not LLM judgment. A mechanical labeler (label_truth_table.mjs) strips comments and strings, resolves entry-point barrels and one-hop export *, and counts references per the shared LABELING_PROTOCOL.md. We chose deterministic over LLM labeling deliberately. An LLM labeler applies a conservative "probably has an out-of-sample consumer" judgment. That is exactly the kind of unverifiable reasoning the polarity guard exists to catch. A mechanical labeler is reproducible. Its biases are inspectable.

With a truth table, the grader no longer asks "was this file supposed to be clean?" It asks a different question. "Of the symbols the detector certified as dead, how many are actually dead?" That is a real precision measurement.

Verdict contract

What contract separates verdicts from candidates?

A per-symbol truth table is necessary but not sufficient. The other half is what the detector is willing to claim.

CD03 has an oracle: a language-graph index that can confirm whether a symbol has a cross-file reference. But the oracle is only as complete as the files it indexed. On a fully-indexed application repo, "no cross-file reference" is a strong signal. On a corpus-sampled package slice, the same answer can be wrong. The consumer lives outside the index.

So the detector now splits its output into two populations:

• Verdict (confidence 0.85): the oracle reports no cross-file reference, and the oracle's own indexed texts agree. The symbol is only declared, not referenced. The detector is willing to stand behind this as "dead."
• Candidate (confidence 0.5): the text heuristic fired but the oracle cannot fully confirm. The index may be partial, or the indexed texts show a reference the top-level index missed. Surfaced for diagnosis, explicitly NOT certified.

Precision is measured on verdicts only. Candidates are diagnostic output, not claims.

Corrected headline

What is the corrected headline?

Scoring oracle-confirmed verdicts against the source-truth table (grade-v3.mjs) confirms the pass:

Precision 0.989, CI entirely above the 0.85 pre-registered floor — a confirmed pass. The single booked FP is zod/regexes.ts:html5Email, an unlabeled symbol conservatively counted as FP. It is in fact genuinely dead. So the true detector-error FP count is 0, and 0.989 is a lower bound.

Recall 0.509 is the measured price of the verdict-only contract on a partially-indexed package. 80 of 84 false negatives are zod verdict_source symbols. The detector surfaced them as candidates but refused to certify them. The sampled language-graph cannot see their out-of-slice consumers. The detector did not miss them — it declined to over-claim. That is the contract working, not a defect. On a fully-indexed application repo (the dogfooding scope) recall is not bounded by this artifact.

v2a vs v3

How does the v2a methodology differ from v3?

The v2a FAIL was produced by scoring correct findings as false positives — genuinely dead zod exports like extendedDuration, uuid4, uuid6, uuid7. The detector was not the problem. The metric was. We retain the naive number as clean_fp_rate_naive for continuity and treat it as methodologically superseded.

This is the same discipline as the Part 3 retraction, applied one level down. Part 3 caught a comparative claim whose CI crossed its own polarity. Part 4 catches a measurement whose ground truth was asserted rather than built. Both are cases where the number looked decisive and wasn't.

Framework

Which three questions decide whether you can trust a precision number?

The correction compresses into three questions any team can ask of a static-analysis precision claim — ours or anyone else's.

1. What is the ground-truth unit, and was it labeled or assumed? Believing "this corpus is clean" makes the precision number measure the belief, not the detector. The unit must match what the detector flags. Here, that is the exported symbol. Each unit must carry an explicit label.

2. What is the detector willing to certify, and what does it only suspect? One undifferentiated stream of findings forces you to score its hunches as claims. Split the output by confidence contract — verdict vs candidate — and score only the claims. The honest precision of the tool is the precision of what it stands behind.

3. Is a "100%" row real, or is it n=0? Five of our six detector classes show precision/recall 1.00 on the real-AI corpus. All five have tp=fp=fn=0 and a degenerate [0.00, 0.00] CI. That is zero statistical power, not accuracy. Report a 100% row with no true positives as "no signal," never as "the detector is perfect on this class."

If a precision claim cannot answer all three, it is a measurement gap, not a finding. Publish the bound, say what is not yet measured, and expand the ground truth before making the directional claim.

Limits

What are the honest limits of this measurement?

The labeler is deterministic, not human. label_truth_table.mjs cannot resolve destructuring or aliasing. It also deliberately skips the "likely has an out-of-sample consumer" conservatism an LLM labeler used on the v2c stratum. Some zod symbols consumed only by out-of-slice packages are labeled verdict_source. That is correct for measuring verdict precision on the sampled slice. Those symbols, when flagged, are true positives by construction. But the labels describe the slice, not the whole package.

Recall is scope-bound. 0.509 is a property of the verdict contract on a corpus-sampled package. It does not generalize to "CD03 has low recall," and we do not claim it does.

The single FP is a policy artifact, not a detector error. Conservative unlabeled→FP accounting books a genuinely dead symbol as a false positive. Reported precision is therefore a lower bound on true verdict precision.

Corpus is TS/JS-centric and partly our own code. zod, express, commander.js, plus our own MCP services. Generalization to Python, Go, or other organizations' style is not claimed.

Detectors

Two more detectors, defined against sources — CD07 and CD08

The same discipline applies to defining a detector, not only to scoring it. A detector defined by intuition ("this feels like dead code") inherits the author's blind spot. So the two newest detectors were defined against external sources first, then implemented and measured.

CD07 — duplicate export. A symbol re-exported from more than one barrel or entry path. This is not our coinage. Knip reports it as the duplicates issue type ("this is exported more than once"). The mcp-use cleanup (#1449) states the mechanism exactly. Symbols "re-exported from more than one entry … force Knip (and humans) to guess which path is canonical." Our suggested action mirrors Knip's fix. Pick the canonical path, collapse the internal re-exports. We score CD07 as a warn (confidence 0.7), not a verdict. The reason is knip #228: a default-plus-named double export can be intentional. Measured on the clean corpus (grade-cd07.mjs): 0 false positives on single-path packages (commander, express). Two true positives in zod, which genuinely re-exports config and lt through parallel public surfaces. Recall 1.0 on a synthetic fan corpus.

CD08 — stale file. A file whose last git commit is older than a threshold (default 90 days). The signal is git last-commit, not filesystem mtime. Three independent tools (git-vision, git-hot, graveyard) all anchor staleness to the commit, because mtime lies under touch, checkout, and formatters. The 90-day default is not a taste number. git-vision's age bands treat 90 days as the conservative lower bound of "stale" territory — its stale band starts at 6 months, with an "aging" band from 3 months, and ancient at 1 year. CodePulse's enterprise branch-hygiene policy auto-archives at 90 days. For per-file staleness 90d is the conservative bound. Tighter thresholds (30–60d) suit branches and PRs but false-positive on intentionally stable modules. Per the spec's no-filesystem rule, the git signal is injected by the caller. The detector stays a pure function and is inert without it. Measured (grade-cd08.mjs): 0 false positives on fresh files. On the mature greg-personal-claude repo (1,101 scripts, real git ages), all 31 files older than 90 days were flagged. Recall 1.0, 0 FP on fresh.

Both are warns, not deletion verdicts. A duplicate export may be deliberate backwards compatibility; a stale file may be load-bearing. The detector surfaces, the human decides — the same verdict-vs-candidate humility as CD03, applied at the definition layer.

Reproducibility

How can you reproduce this measurement?

Six steps, all local, all deterministic at $0 marginal cost:

1. Build the detector bundle the grader imports: cd mcp/source/services/code-diet-mcp && npm run build && cd -.
2. Run unit tests (detectors + the language-graph oracle + findings store): node --test mcp/source/services/code-diet-mcp/test/detectors.test.mjs and node --test mcp/source/services/code-diet-mcp/test/findings_store.test.mjs.
3. Rebuild the corpora (idempotent given the seed) from benchmarks/code-diet/: node scripts/build-clean-corpus.mjs, node scripts/build-injected-corpus.mjs, node scripts/build-real-ai-corpus.mjs.
4. Rebuild the v3 ground truth — the deterministic per-symbol truth table: node label_truth_table.mjs → truth_table_v3.json + truth_table_v3.md.
5. Run the v3 grader — verdict-only scoring against the source-truth table: node grade-v3.mjs → results/eval_v3_.json (continuity: node grade-v2.mjs for the naive v2a accounting).
6. Run the CD07/CD08 graders: node grade-cd07.mjs → results/eval_cd07_.json and node grade-cd08.mjs → results/eval_cd08_.json.

The detector, grader, labeler, truth table, and raw result JSONs live in the eval harness. The de-tuned public baseline of the detectors ships in the code-quality repo for independent replication.

Working directory: benchmarks/code-diet/ unless noted.

# 1. Build the detector bundle the grader imports
cd mcp/source/services/code-diet-mcp && npm run build && cd -

# 2. Unit tests (detectors + the language-graph oracle + findings store)
node --test mcp/source/services/code-diet-mcp/test/detectors.test.mjs
node --test mcp/source/services/code-diet-mcp/test/findings_store.test.mjs

# 3. Rebuild the corpora (idempotent given the seed)
cd benchmarks/code-diet
node scripts/build-clean-corpus.mjs
node scripts/build-injected-corpus.mjs
node scripts/build-real-ai-corpus.mjs

# 4. Deterministic per-symbol truth table (v3 ground truth)
node label_truth_table.mjs
# -> truth_table_v3.json + truth_table_v3.md

# 5. Grader v3 — verdict precision on source truth (the corrected headline)
node grade-v3.mjs
# -> results/eval_v3_<date>.json

# 6. Grader v2 — naive per-class P/R/F1 + FP-rate (retained for continuity)
node grade-v2.mjs

# 7. CD07 duplicate-export + CD08 stale-file graders
node grade-cd07.mjs   # -> results/eval_cd07_<date>.json
node grade-cd08.mjs   # -> results/eval_cd08_<date>.json

Closing

The number looked decisive, and the metric was the bug.

A 38%-precision FAIL and a 98.9%-precision PASS, on the same detector, with no code change in between. The difference was not tuning. It was refusing to score a blanket assumption as ground truth. It was refusing to let a partial graph certify "dead."

This is the through-line of the series. Part 1 established the measurement infrastructure. Part 2 built the statistics. Part 3 retracted a comparative claim whose CI crossed its own polarity. Part 4 retracts a measurement whose ground truth was asserted rather than built. The discipline is the same. A number you cannot defend is worse than no number. It anchors every decision downstream of it.

If you ship a static-analysis detector and have never built a per-symbol truth table for it, your precision claim measures your belief about the corpus. Build the table. Split verdicts from candidates. Then publish the bound.

Honest framing note. I built the code-diet detector, the language-graph oracle, the corpus, the labeler, and the grader. That combination of roles is a known source of bias. It is the bias the blind-labeling and polarity-guard rules exist to contain. The detector source, the deterministic labeler, the truth table, and the raw result JSONs live in the eval harness and the public code-quality repo. Any reader can rebuild the truth table and recompute the precision number on their own checkout. When I report "0.989 precision," I am reporting an effect measured on one corpus against one deterministic labeling of one package slice. The v2a→v3 correction in this article is exactly that risk firing — and being caught by the methodology rather than by a reader. Replicating with other corpora, other labelers, and ideally another evaluator turns this into a general claim rather than my own claim about my own infrastructure.

FAQ

Frequently asked questions

Sources

Sources

Related reading

Where should you go next?