DarkMoon research series·D. Open source & licence·Article 20 of 23·All 23 articles
Introduction
DarkMoon is an autonomous AI penetration-testing system: an orchestrator and specialist LLM agents run tools against an authorised target and publish findings. Claims about such systems are easy to make and hard to check. We publish our research as public records, with the failures kept in, so that readers can check our numbers instead of trusting them.
This article describes what is published, how it is labelled, and where reproducibility stops. It is the last article of this series and links both research repositories:
- ASCIT31/darkmoon-attack-paths-research: how DarkMoon links findings into attack paths, and 61 experiment pages;
- ASCIT31/darkmoon-visualization-research: how results are shown, in the web dashboard and the terminal.
Problem
Three failure modes are common in published claims about AI security tools, and we have made all three ourselves at some point:
- Single runs presented as results. Our own Community README still headlines « 57 findings in 28.5 min » on OWASP Juice Shop. That is a single run, with no sealed ground truth. The attack-paths research repository records it as
UNKNOWN(EXP-004). - Definitions that move. On 11 November 2026 an internal report described « two genuinely proven multi-step cross-host attack chains ». Under the stricter definition adopted later, those chains were agent-declared, not proven. The claim is now marked
SUPERSEDED, and the contradiction is recorded rather than deleted. - Silent omission of what was not measured. Absence of a measurement tends to read as absence of a problem.
Methodology
Every statement carries an evidence label
Both research repositories use the same six labels:
| Label | Meaning |
|---|---|
VERIFIED | checked in a primary artefact (code, git object, build log, or a report cross-checked with a second artefact) |
OBSERVED | stated in one primary source, not re-executed |
SIMULATED | deterministic harness, replay, synthetic corpus or proxy model, never a real-run precision |
HYPOTHESIS | an interpretation, or a claim no source tested |
SUPERSEDED | replaced by later work, kept for history |
UNKNOWN | the evidence was not available |
Failures are first-class results
The attack-paths registry has 61 experiments (2 July → 20 November 2026). By outcome: 17 successful, 19 failed, 17 inconclusive, 8 adversarial. By label: 18 VERIFIED, 20 SIMULATED, 12 OBSERVED, 9 SUPERSEDED, 2 UNKNOWN (VERIFIED, registry index). More experiments failed than succeeded, and they are published on the same footing.
Two failed results that shaped the product:
- A MulVAL-like capability graph built from vulnerability taxonomies linked findings correctly in 10 of 78 edges (12.8 %) (
SUPERSEDED, EXP-039). - On a real multi-protocol campaign, agent-declared links had a strict precision of 0.316 [0.15–0.54], and 0 of 19 were qualified by the server (
VERIFIED, n = 1, EXP-050).
On the visualization side, the failures page lists rejected UI directions with the owner's verdict and the reverting commits (see Our UI Experiments).
Reading rules travel with the numbers
The experiment pages carry binding reading rules: never compare finding counts across targets; never read a harness score as agent precision; n = 1 almost everywhere; « proven » changed meaning over time; costs are USD actually billed as recorded. Wilson intervals (Wilson, 1927) are given where the source gives them, and most real-run comparisons overlap.
A « not measured » list is part of the result
The visualization repository's benchmarks page reports the measures that exist and then lists those that do not:
| Exists | Does not exist (not measured) |
|---|---|
| production initial bundle 1.23–1.26 MB raw in three build logs (VERIFIED) | Lighthouse on the Angular client |
| dashboard CLS 0.1065 → fixed (VERIFIED) | frame rate or GPU cost of the 3D map; ELK layout time |
| Terminal Explorer: 0 model calls while browsing, +44 MB RSS (VERIFIED) | terminal render time |
| test counts per iteration (VERIFIED) | any user study: task time, error rate, SUS, comprehension of proof labels |
Benchmarks that can be recomputed
Detection results live in a third repository, ASCIT31/Darkmoon-Benchmarks. Its 2026 study covers 21 runs of 16 tool × model configurations on OWASP Juice Shop. Every stored score is recomputed from per-challenge rows by standard-library scripts, which also run in CI. Findings from that study, as published there:
- the best valid runs solved 16 challenges and about 11 % of the catalogue's difficulty points (the DarkMoon CLI configuration scored 43 points; another configuration scored 42);
- the solve rate falls with difficulty, from 13.5 % at 1 star to 0 % at 6 stars;
- repeated runs of the same configuration solve different challenges (Jaccard 0.00 to 0.60), so single-run rankings are unstable;
- only one configuration records its tool, and only one run has metered cost.
The repository states that every run was executed by the DarkMoon team and that none of it is an independent evaluation. Detection figures (findings or challenges per run) and link figures (precision of declared edges) are different metrics and must not be combined.
Real results: what others can and cannot reproduce
| Can be reproduced or checked by anyone | Cannot be reproduced today |
|---|---|
Benchmark scores from the published dataset (validate_results.py, generate_tables.py --check) | the benchmark runs themselves (unknown tools for most configurations, unpinned target digest, prompts not recorded) |
Community code paths cited at a fixed commit (10eaab0), e.g. the attack-path engine and the Terminal Explorer | the internal reports (R/nn, IR-nn) that many pages cite; they are private and cited by identifier for traceability |
| The property « Community never proves a path », in code and on any Community campaign | DarkMoon Pro behaviour: Pro is proprietary, cited by short commit hash only |
UI captures with commit, viewport, theme and data source in metadata.json | most real-campaign figures: n = 1, one target, one model (claude-opus-4-6) |
| Licensing state of the Community repository (licence file history, release list) | the internal harnesses and oracles used for edge-level scoring (described, not released) |

An example of a published limit in the product itself: without proven paths, the plan says what it ranks by instead of ranking by paths. Real TUI, synthetic demo data.
Limits
- Self-assessment. The research records were assembled by the team that built the system. Oracles and harnesses were built by the same programme; some replays used a proxy model.
- Targets are well known. Juice Shop, DVGA and crAPI are likely present in model training data; Juice Shop runs of one programme were excluded after contamination.
- Our own product text still overstates in places. The research repositories list them (for example the README benchmark headline and two attack-path strings) and keep the correction pending rather than hiding the discrepancy.
- Openness is partial. The Pro source code, internal reports and harnesses are not public. We publish their identifiers, results and limits, not their content.
- No external replication exists yet of any figure in these repositories.
Figures
- Figure 1 (Mermaid): publication pipeline.
- Capture: Community Terminal Explorer, from the attack-paths research repository's
screenshots/(withmetadata.json).

Scientific sources (verified)
- E. B. Wilson. « Probable Inference, the Law of Succession, and Statistical Inference ». Journal of the American Statistical Association 22(158), 1927. doi:10.1080/01621459.1927.10502953
- A. K. Zhang et al. « Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models ». ICLR 2025. arXiv:2408.08926
- J. Pearl. Causality: Models, Reasoning, and Inference. 2nd ed., Cambridge University Press, 2009. doi:10.1017/CBO9780511803161
- W3C. PROV-DM: The PROV Data Model, 2013. https://www.w3.org/TR/2013/REC-prov-dm-20130430/
DOIs checked against Crossref, other URLs fetched, on 2026-11-20. The full verified bibliographies are in each repository.
Research, source code & reproducibility
- Main research repositories: ASCIT31/darkmoon-attack-paths-research (experiments, RESULTS.md, limitations) and ASCIT31/darkmoon-visualization-research (benchmarks, experiments, screenshots). Both include a
CITATION.cff. - Source code: ASCIT31/Dark-Moon (Community, AGPL-3.0 on
master; Terminal Explorer atconf/plugins/darkmoon-explorer). - Benchmarks: ASCIT31/Darkmoon-Benchmarks (code GPL-3.0, data CC BY 4.0, per its README).
- Documentation: https://docs.dark-moon.org
- References: references.bib · visualization references.
- Reproducible experiments:
git clone https://github.com/ASCIT31/Darkmoon-Benchmarks.git cd Darkmoon-Benchmarks/studies/llm-pentest-juice-shop-2026 python3 scripts/validate_results.py python3 scripts/generate_tables.py --check
Related reading: Our UI Experiments: What Worked, What Failed and Why · Measuring Attack-Path Reliability: Benchmarks, False Positives and Limitations · Why Our First Attack-Path Architectures Failed · Designing Explainable Security Graphs Without Misrepresenting Evidence · Why DarkMoon Community Is Moving from GPLv3 to AGPLv3
If you replicate, contest or extend any figure in these repositories, open an issue on the relevant one. A failed replication is as welcome as a successful one.
See the proof, not just the write-up: the Pro remediation benchmark (fixes retested against the exploit) · how Darkmoon compares to other AI pentest tools.
← All articles