Blog··7 min read

Measuring Attack-Path Reliability: Benchmarks, False Positives and Limitations

Harnesses, mutation tests, replays and real campaigns: how DarkMoon measures attack-path link quality, the numbers, the intervals, and what is not conclusive.

DarkMoon research series·A. Architecture & attack paths·Article 7 of 23·All 23 articles

Introduction

Autonomous pentest tools are usually evaluated on detection: how many vulnerabilities, how many flags, how fast. Attack paths need a different question: when the system links two findings, is the link right? Detection benchmarks cannot answer it, and, as far as the verified literature reviewed by the DarkMoon team goes, no public edge-level benchmark exists.

This article describes the instruments DarkMoon used to measure link quality, the figures they produced, and the reading rules that keep those figures honest. Everything cited is in the attack-paths research repository.

The problem

Three things make link quality hard to measure:

  1. Ground truth is per campaign. Whether finding A enabled finding B depends on what the agent actually did in that run. Someone has to adjudicate it.
  2. Runs are expensive and noisy. A real campaign cost between about USD 6 and 20 in this programme, and two runs with similar prompts can differ a lot.
  3. Many convenient proxies measure something else. A deterministic harness measures the verifier, not the agent. A proxy model measures a different model. A detection count measures findings, not links.

Methodology

Seven instruments, each with a scope

InstrumentCan showCannot show
Sealed deterministic harness with an independent oraclethe engine never promotes on adversarial casesreal-world precision or recall of agents
Mutation testingthe tests constrain the rulesthat the rules fit real data
Replay of stored real campaignswhat the current engine would decide on past datawhat agents would declare under new prompts
Proxy-model simulationthe direction of prompt effects on given transcriptsthe campaign model's behaviour
Real campaign plus independent oracledeclared-link precision and recall on that target, n = 1generalisation
Counterfactual replaynecessity for specific candidates on a labanything without lab-side references; it is active traffic
Blind adjudicationthe truth of specific claimscoverage

Five binding reading rules

  1. Never compare finding counts across targets (Juice Shop, crAPI, DVGA, the internal lab and the multi-protocol lab differ in size and content).
  2. Never read a harness or proxy score as agent-declaration precision; those rows are SIMULATED.
  3. n = 1 almost everywhere; Wilson intervals are given where the source gives them.
  4. "Proven" changed meaning over time; older figures use the definition in force at the time.
  5. Costs are USD actually billed, as recorded.

Scoring

Per-campaign oracles were hand-adjudicated with strict and lenient scoring. The headline is strict precision of declared links: a declared link counts as correct only if it matches the direct parent in the oracle.

Real results

The verifier, on synthetic scenarios (SIMULATED)

  • Architecture D harness: 89, then 100 scenarios (102, then 117 edges); 0 false promotions; recall 0.829, then 0.867.
  • Mutation score 90.5 % → 94.4 %, then, after a run that exposed 38 permissive survivors, 95.2 % with 0 permissive survivors.
  • Six "major" missed promotions in the final review were judged prudent abstentions.

The agents, on real campaigns (VERIFIED unless marked)

CampaignFindingsDeclared linksStrict precisionStrict recallServer-qualified
c5 (Juice Shop)39160.6 (5 decidable)2/60/16 (replay, SIMULATED)
c6 (multi-protocol lab)1190.556 [0.267 to 0.811]5/60/9 (replay, SIMULATED)
c7 (multi-protocol lab)28190.316 [0.154 to 0.540]6/90/19
final lab run2919≈ 0.37 (approximate)—0/19

Costs: c5 USD 15.11 (budget cap), c6 USD 6.05, c7 USD 9.27, final run USD 8.46.

Things that move the numbers

  • Oracle sensitivity. One mapping choice on c7 (counting PostgreSQL access and superuser as separate types) moves strict precision by about 10 points, to 42 %.
  • Variance. Proxy-model passes of the same prompt variant on the same corpus differed by about ±20 points. Real cost rose 53 % between c6 and c7 with similar prompts.
  • Contamination. Two Juice Shop runs of the R&D programme were excluded: the agent made 5 and 35 attempts on the target's own scoring endpoints (EXP-016).
  • Metadata disagreement. On the final lab run, the KPI file reports no budget-guard stop while the guard log reports one. The repository records the contradiction.

False positives, by class

On c7, 12 of 13 false links were "same root cause" facets (4 Redis facets, 4 version-matched CVEs, 2 SMB facets, 2 state findings), all from sub-agents; c6's false positives were 3 transitive (an ancestor cited instead of the parent) and 1 not-a-path. A wording change eliminated transitive errors (3 → 0) but the new error class appeared, so the net effect was not attributable and the change was reverted.

Statistical conclusion

No pair of real runs in this programme separates two configurations with non-overlapping Wilson intervals. A behavioural A/B test would need at least five real campaigns per variant (about USD 60); it has not been run. The honest summary is "not conclusive".

Detection is a separate question

DarkMoon's public detection records live in Darkmoon-Benchmarks, with their own method and stated limits. Detection figures and link figures must not be combined. The "57 findings in 28.5 minutes on Juice Shop" headline in the main README is a single run without sealed ground truth; the attack-paths repository records it as UNKNOWN (EXP-004).

Limits

  • One campaign model throughout (claude-opus-4-6); one adjudicator per campaign; oracles and harnesses built by the same team.
  • Targets are self-built labs or well-known vulnerable applications likely present in training data; no run on a public benchmark.
  • Several replays used Claude Code sub-agents as a proxy model, with a different model, a shorter context and a neutral prompt; the proxy did not reproduce the real same-cause errors on synthetic transcripts.
  • The internal harnesses and oracles are not released; their results are described on the experiment pages.

Figures

Diagram. Figure 1. Point estimates only. Wilson intervals: c6 0.27 to 0.81, c7 0.15 to 0.54; c5 rests on 5 decidable links. The intervals overlap, so the chart does not show a trend.
Figure 1. Point estimates only. Wilson intervals: c6 0.27 to 0.81, c7 0.15 to 0.54; c5 rests on 5 decidable links. The intervals overlap, so the chart does not show a trend.
Diagram. Figure 2. Which instrument measures which property. None of them measures cross-target generalisation.
Figure 2. Which instrument measures which property. None of them measures cross-target generalisation.

Scientific sources

  • Kapoor, Stroebl, Siegel et al. AI Agents That Matter. 2024. arXiv:2407.01502
  • Xu et al. Benchmark Data Contamination of Large Language Models: A Survey. 2024. arXiv:2406.04244
  • Zhang et al. Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models. 2025. arXiv:2408.08926
  • Gioacchini et al. AutoPenBench: Benchmarking Generative Agents for Penetration Testing. 2024. arXiv:2410.03225
  • Peng et al. Hackers or Hallucinators? A Comprehensive Analysis of LLM-Based Automated Penetration Testing. 2026. arXiv:2604.05719
  • Bouchari et al. Autonomous LLM Agents & CTFs: A Second Look. 2026. arXiv:2605.21497

Research, source code & reproducibility


The first open research direction listed in the repository is an edge-level benchmark with sealed truth, decoy common-cause pairs and released traces. If you are working on something similar, comparing protocols early would save both teams a lot of money.

Next
Run it against your own lab

Darkmoon is open source (AGPL-3.0) and self hosted. Clone it, point it at a target you own, and read every line.