Blog··8 min read

Building DarkMoon in the Open: Research, Benchmarks, Failures and Reproducibility

How DarkMoon publishes its research: 61 experiments incl. 19 failed, evidence labels, a benchmark built to be recomputed, and what cannot be reproduced.

DarkMoon research series·D. Open source & licence·Article 20 of 23·All 23 articles

Introduction

DarkMoon is an autonomous AI penetration-testing system: an orchestrator and specialist LLM agents run tools against an authorised target and publish findings. Claims about such systems are easy to make and hard to check. We publish our research as public records, with the failures kept in, so that readers can check our numbers instead of trusting them.

This article describes what is published, how it is labelled, and where reproducibility stops. It is the last article of this series and links both research repositories:

Problem

Three failure modes are common in published claims about AI security tools, and we have made all three ourselves at some point:

  1. Single runs presented as results. Our own Community README still headlines « 57 findings in 28.5 min » on OWASP Juice Shop. That is a single run, with no sealed ground truth. The attack-paths research repository records it as UNKNOWN (EXP-004).
  2. Definitions that move. On 11 November 2026 an internal report described « two genuinely proven multi-step cross-host attack chains ». Under the stricter definition adopted later, those chains were agent-declared, not proven. The claim is now marked SUPERSEDED, and the contradiction is recorded rather than deleted.
  3. Silent omission of what was not measured. Absence of a measurement tends to read as absence of a problem.

Methodology

Every statement carries an evidence label

Both research repositories use the same six labels:

LabelMeaning
VERIFIEDchecked in a primary artefact (code, git object, build log, or a report cross-checked with a second artefact)
OBSERVEDstated in one primary source, not re-executed
SIMULATEDdeterministic harness, replay, synthetic corpus or proxy model, never a real-run precision
HYPOTHESISan interpretation, or a claim no source tested
SUPERSEDEDreplaced by later work, kept for history
UNKNOWNthe evidence was not available

Failures are first-class results

The attack-paths registry has 61 experiments (2 July → 20 November 2026). By outcome: 17 successful, 19 failed, 17 inconclusive, 8 adversarial. By label: 18 VERIFIED, 20 SIMULATED, 12 OBSERVED, 9 SUPERSEDED, 2 UNKNOWN (VERIFIED, registry index). More experiments failed than succeeded, and they are published on the same footing.

Two failed results that shaped the product:

  • A MulVAL-like capability graph built from vulnerability taxonomies linked findings correctly in 10 of 78 edges (12.8 %) (SUPERSEDED, EXP-039).
  • On a real multi-protocol campaign, agent-declared links had a strict precision of 0.316 [0.15–0.54], and 0 of 19 were qualified by the server (VERIFIED, n = 1, EXP-050).

On the visualization side, the failures page lists rejected UI directions with the owner's verdict and the reverting commits (see Our UI Experiments).

Reading rules travel with the numbers

The experiment pages carry binding reading rules: never compare finding counts across targets; never read a harness score as agent precision; n = 1 almost everywhere; « proven » changed meaning over time; costs are USD actually billed as recorded. Wilson intervals (Wilson, 1927) are given where the source gives them, and most real-run comparisons overlap.

A « not measured » list is part of the result

The visualization repository's benchmarks page reports the measures that exist and then lists those that do not:

ExistsDoes not exist (not measured)
production initial bundle 1.23–1.26 MB raw in three build logs (VERIFIED)Lighthouse on the Angular client
dashboard CLS 0.1065 → fixed (VERIFIED)frame rate or GPU cost of the 3D map; ELK layout time
Terminal Explorer: 0 model calls while browsing, +44 MB RSS (VERIFIED)terminal render time
test counts per iteration (VERIFIED)any user study: task time, error rate, SUS, comprehension of proof labels

Benchmarks that can be recomputed

Detection results live in a third repository, ASCIT31/Darkmoon-Benchmarks. Its 2026 study covers 21 runs of 16 tool × model configurations on OWASP Juice Shop. Every stored score is recomputed from per-challenge rows by standard-library scripts, which also run in CI. Findings from that study, as published there:

  • the best valid runs solved 16 challenges and about 11 % of the catalogue's difficulty points (the DarkMoon CLI configuration scored 43 points; another configuration scored 42);
  • the solve rate falls with difficulty, from 13.5 % at 1 star to 0 % at 6 stars;
  • repeated runs of the same configuration solve different challenges (Jaccard 0.00 to 0.60), so single-run rankings are unstable;
  • only one configuration records its tool, and only one run has metered cost.

The repository states that every run was executed by the DarkMoon team and that none of it is an independent evaluation. Detection figures (findings or challenges per run) and link figures (precision of declared edges) are different metrics and must not be combined.

Diagram. Figure 1. From internal evidence to public records.
Figure 1. From internal evidence to public records.

Real results: what others can and cannot reproduce

Can be reproduced or checked by anyoneCannot be reproduced today
Benchmark scores from the published dataset (validate_results.py, generate_tables.py --check)the benchmark runs themselves (unknown tools for most configurations, unpinned target digest, prompts not recorded)
Community code paths cited at a fixed commit (10eaab0), e.g. the attack-path engine and the Terminal Explorerthe internal reports (R/nn, IR-nn) that many pages cite; they are private and cited by identifier for traceability
The property « Community never proves a path », in code and on any Community campaignDarkMoon Pro behaviour: Pro is proprietary, cited by short commit hash only
UI captures with commit, viewport, theme and data source in metadata.jsonmost real-campaign figures: n = 1, one target, one model (claude-opus-4-6)
Licensing state of the Community repository (licence file history, release list)the internal harnesses and oracles used for edge-level scoring (described, not released)
Prioritized Treatment Plan in the Community Terminal Explorer at 160×45, ordered by reported severity and proof because no Evidence-Backed Attack Path exists; the action panel lists related paths: 0 Evidence-Backed, 2 Possible.
Prioritized Treatment Plan in the Community Terminal Explorer at 160×45, ordered by reported severity and proof because no Evidence-Backed Attack Path exists; the action panel lists related paths: 0 Evidence-Backed, 2 Possible.Original capture: ASCIT31/darkmoon-attack-paths-research

An example of a published limit in the product itself: without proven paths, the plan says what it ranks by instead of ranking by paths. Real TUI, synthetic demo data.

Limits

  • Self-assessment. The research records were assembled by the team that built the system. Oracles and harnesses were built by the same programme; some replays used a proxy model.
  • Targets are well known. Juice Shop, DVGA and crAPI are likely present in model training data; Juice Shop runs of one programme were excluded after contamination.
  • Our own product text still overstates in places. The research repositories list them (for example the README benchmark headline and two attack-path strings) and keep the correction pending rather than hiding the discrepancy.
  • Openness is partial. The Pro source code, internal reports and harnesses are not public. We publish their identifiers, results and limits, not their content.
  • No external replication exists yet of any figure in these repositories.

Figures

  • Figure 1 (Mermaid): publication pipeline.
  • Capture: Community Terminal Explorer, from the attack-paths research repository's screenshots/ (with metadata.json).
DarkMoon Pro campaign overview of an anonymised real campaign against OWASP Juice Shop (26 April 2026, 57 findings): the campaign header with Map, Attack paths, Findings, Overview and Reports tabs, a Cyber Score of E 0/100 marked Rated with an explanation of what holds the grade, and the first recommended actions (access control, authentication).
Figure 2. The same habit in DarkMoon Pro, the proprietary edition: the campaign overview states how much was tested and what holds the Cyber Score before it lists the first actions. Anonymised real campaign against OWASP Juice Shop, an intentionally vulnerable training application (demo.dark-moon.org).Captured 21 November 2026 from demo.dark-moon.org (current interface).

Scientific sources (verified)

DOIs checked against Crossref, other URLs fetched, on 2026-11-20. The full verified bibliographies are in each repository.

Research, source code & reproducibility

Related reading: Our UI Experiments: What Worked, What Failed and Why · Measuring Attack-Path Reliability: Benchmarks, False Positives and Limitations · Why Our First Attack-Path Architectures Failed · Designing Explainable Security Graphs Without Misrepresenting Evidence · Why DarkMoon Community Is Moving from GPLv3 to AGPLv3


If you replicate, contest or extend any figure in these repositories, open an issue on the relevant one. A failed replication is as welcome as a successful one.

Next
Run it against your own lab

Darkmoon is open source (AGPL-3.0) and self hosted. Clone it, point it at a target you own, and read every line.