DarkMoon research series·A. Architecture & attack paths·Article 7 of 23·All 23 articles
Introduction
Autonomous pentest tools are usually evaluated on detection: how many vulnerabilities, how many flags, how fast. Attack paths need a different question: when the system links two findings, is the link right? Detection benchmarks cannot answer it, and, as far as the verified literature reviewed by the DarkMoon team goes, no public edge-level benchmark exists.
This article describes the instruments DarkMoon used to measure link quality, the figures they produced, and the reading rules that keep those figures honest. Everything cited is in the attack-paths research repository.
The problem
Three things make link quality hard to measure:
- Ground truth is per campaign. Whether finding A enabled finding B depends on what the agent actually did in that run. Someone has to adjudicate it.
- Runs are expensive and noisy. A real campaign cost between about USD 6 and 20 in this programme, and two runs with similar prompts can differ a lot.
- Many convenient proxies measure something else. A deterministic harness measures the verifier, not the agent. A proxy model measures a different model. A detection count measures findings, not links.
Methodology
Seven instruments, each with a scope
| Instrument | Can show | Cannot show |
|---|---|---|
| Sealed deterministic harness with an independent oracle | the engine never promotes on adversarial cases | real-world precision or recall of agents |
| Mutation testing | the tests constrain the rules | that the rules fit real data |
| Replay of stored real campaigns | what the current engine would decide on past data | what agents would declare under new prompts |
| Proxy-model simulation | the direction of prompt effects on given transcripts | the campaign model's behaviour |
| Real campaign plus independent oracle | declared-link precision and recall on that target, n = 1 | generalisation |
| Counterfactual replay | necessity for specific candidates on a lab | anything without lab-side references; it is active traffic |
| Blind adjudication | the truth of specific claims | coverage |
Five binding reading rules
- Never compare finding counts across targets (Juice Shop, crAPI, DVGA, the internal lab and the multi-protocol lab differ in size and content).
- Never read a harness or proxy score as agent-declaration precision; those rows are
SIMULATED. - n = 1 almost everywhere; Wilson intervals are given where the source gives them.
- "Proven" changed meaning over time; older figures use the definition in force at the time.
- Costs are USD actually billed, as recorded.
Scoring
Per-campaign oracles were hand-adjudicated with strict and lenient scoring. The headline is strict precision of declared links: a declared link counts as correct only if it matches the direct parent in the oracle.
Real results
The verifier, on synthetic scenarios (SIMULATED)
- Architecture D harness: 89, then 100 scenarios (102, then 117 edges); 0 false promotions; recall 0.829, then 0.867.
- Mutation score 90.5 % → 94.4 %, then, after a run that exposed 38 permissive survivors, 95.2 % with 0 permissive survivors.
- Six "major" missed promotions in the final review were judged prudent abstentions.
The agents, on real campaigns (VERIFIED unless marked)
| Campaign | Findings | Declared links | Strict precision | Strict recall | Server-qualified |
|---|---|---|---|---|---|
| c5 (Juice Shop) | 39 | 16 | 0.6 (5 decidable) | 2/6 | 0/16 (replay, SIMULATED) |
| c6 (multi-protocol lab) | 11 | 9 | 0.556 [0.267 to 0.811] | 5/6 | 0/9 (replay, SIMULATED) |
| c7 (multi-protocol lab) | 28 | 19 | 0.316 [0.154 to 0.540] | 6/9 | 0/19 |
| final lab run | 29 | 19 | ≈ 0.37 (approximate) | — | 0/19 |
Costs: c5 USD 15.11 (budget cap), c6 USD 6.05, c7 USD 9.27, final run USD 8.46.
Things that move the numbers
- Oracle sensitivity. One mapping choice on c7 (counting PostgreSQL access and superuser as separate types) moves strict precision by about 10 points, to 42 %.
- Variance. Proxy-model passes of the same prompt variant on the same corpus differed by about ±20 points. Real cost rose 53 % between c6 and c7 with similar prompts.
- Contamination. Two Juice Shop runs of the R&D programme were excluded: the agent made 5 and 35 attempts on the target's own scoring endpoints (EXP-016).
- Metadata disagreement. On the final lab run, the KPI file reports no budget-guard stop while the guard log reports one. The repository records the contradiction.
False positives, by class
On c7, 12 of 13 false links were "same root cause" facets (4 Redis facets, 4 version-matched CVEs, 2 SMB facets, 2 state findings), all from sub-agents; c6's false positives were 3 transitive (an ancestor cited instead of the parent) and 1 not-a-path. A wording change eliminated transitive errors (3 → 0) but the new error class appeared, so the net effect was not attributable and the change was reverted.
Statistical conclusion
No pair of real runs in this programme separates two configurations with non-overlapping Wilson intervals. A behavioural A/B test would need at least five real campaigns per variant (about USD 60); it has not been run. The honest summary is "not conclusive".
Detection is a separate question
DarkMoon's public detection records live in Darkmoon-Benchmarks, with their own method and stated limits. Detection figures and link figures must not be combined. The "57 findings in 28.5 minutes on Juice Shop" headline in the main README is a single run without sealed ground truth; the attack-paths repository records it as UNKNOWN (EXP-004).
Limits
- One campaign model throughout (claude-opus-4-6); one adjudicator per campaign; oracles and harnesses built by the same team.
- Targets are self-built labs or well-known vulnerable applications likely present in training data; no run on a public benchmark.
- Several replays used Claude Code sub-agents as a proxy model, with a different model, a shorter context and a neutral prompt; the proxy did not reproduce the real same-cause errors on synthetic transcripts.
- The internal harnesses and oracles are not released; their results are described on the experiment pages.
Figures
Scientific sources
- Kapoor, Stroebl, Siegel et al. AI Agents That Matter. 2024. arXiv:2407.01502
- Xu et al. Benchmark Data Contamination of Large Language Models: A Survey. 2024. arXiv:2406.04244
- Zhang et al. Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models. 2025. arXiv:2408.08926
- Gioacchini et al. AutoPenBench: Benchmarking Generative Agents for Penetration Testing. 2024. arXiv:2410.03225
- Peng et al. Hackers or Hallucinators? A Comprehensive Analysis of LLM-Based Automated Penetration Testing. 2026. arXiv:2604.05719
- Bouchari et al. Autonomous LLM Agents & CTFs: A Second Look. 2026. arXiv:2605.21497
Research, source code & reproducibility
- Main research repository: ASCIT31/darkmoon-attack-paths-research, see experiments/RESULTS.md and benchmarks/
- Source code (open source): ASCIT31/Dark-Moon
- Benchmarks: ASCIT31/Darkmoon-Benchmarks (detection records, reference lab OWASP Juice Shop)
- Official documentation: docs.dark-moon.org
- Scientific references: literature review §3.9 evaluation
- Reproducible experiments: EXP-042, EXP-043, EXP-047, EXP-049, EXP-050, EXP-053, EXP-056
Related articles
- From Vulnerability Findings to Attack Paths: DarkMoon's Research Journey
- Why Our First Attack-Path Architectures Failed
- Declared vs Proven Attack Paths: A Practical Evidence Model
- Cross-Agent Dependencies and the Common-Root-Cause Problem
- Building DarkMoon in the Open: Research, Benchmarks, Failures and Reproducibility
The first open research direction listed in the repository is an edge-level benchmark with sealed truth, decoy common-cause pairs and released traces. If you are working on something similar, comparing protocols early would save both teams a lot of money.
See the proof, not just the write-up: the Pro remediation benchmark (fixes retested against the exploit) · how Darkmoon compares to other AI pentest tools.
← All articles