Blog··9 min read

Why Our First Attack-Path Architectures Failed

Topology paths, structured claims, log attribution, counterfactual replay and execution ids: why each DarkMoon attack-path design was rejected, with the data.

DarkMoon research series·A. Architecture & attack paths·Article 5 of 23·All 23 articles

Introduction

Negative results are rarely published, and engineering teams rarely keep them. DarkMoon kept them: 19 of the 61 experiment pages in the attack-paths research repository are classed as failed, and 9 entries are marked SUPERSEDED.

This article walks through the attack-path designs that did not survive, in chronological order. Each one was reasonable when it was proposed. Each one failed for a reason that is useful to anyone building attack paths on top of automated or AI-driven testing.

The problem

Every design below tried to answer the same question: given the findings of a campaign, which ones are linked by an actual exploitation step? The common trap was that a design can produce many links, or links that look rigorous, while the links are not true. The bar the team converged on was simple to state and hard to meet: no link may be shown as proven unless the system itself recorded the evidence for it.

Methodology

For each design the repository records what was tried, why, what happened, the decision and the sources. Statements carry evidence labels; deterministic replays and harnesses are SIMULATED; real campaigns with recorded artefacts are VERIFIED. Below, each failure is summarised with its root cause, which is the part that transfers.

Real results: six failures and what caused them

1. Topology as attack paths (September 2026)

Breadth-first paths over the infrastructure graph, plus findings attached to nodes. On 18 real campaigns the server returned 43 topological paths, and the interface read none of them; 22 % of findings were attached through a fallback, up to 31 of 35 findings on one node (VERIFIED).

Root cause. Reachability is not exploitation. A topological path says nothing about whether any step was taken. It survives only as the map and as a labelled inferred layer, capped at 40 hypotheses per campaign and never ranked.

2. Structured causal claims (late October 2026)

Four optional fields on each finding (prerequisites, produced artefacts, reached assets, negative control). Result on 18 campaigns: 0 of 340 findings carried any causal field; 0 of 18 finalisation blocks; ranking allowed on 0 of 18 campaigns (EXP-010, VERIFIED).

Root cause. 50 of 51 pushing prompts never asked specialists to fill the fields, and pivot evidence was truncated or masked anyway. A schema that agents are not told to use stays empty.

3. Attributing findings to observed commands (November 9 to 16)

A run ledger recorded, with salted fingerprints only, what each command produced and consumed. On top of it, an "own command" rule said a finding owns the observed command whose distinctive values it reuses. Deterministic probes found 5 families of false "proven" links, then, after fixes, 7 new false links from skeleton collisions (EXP-038). The offline provenance replay found 1 of 60 truth pairs on development data and 0 of 15 on held-out data (EXP-017, SIMULATED).

Root cause. Coarse dependency produces false dependency, the same lesson OS-level provenance research learned from dependency explosion. The branch was frozen; the ledger was kept, its autonomous attribution deleted.

4. Counterfactual and passive verification (November 8 to 11)

A preregistered programme tried counterfactual active replay (four rounds), a passive evidence engine, active replay on crAPI and a capability graph. Every approach failed its preregistered criterion (VERIFIED):

ApproachResult
Counterfactual replay, round 19/11 correct, 2 false proven links; wire flows 20/20 but recall 4/29 (SIMULATED)
Round 234/40 correct, 6 false, all confirmed false by blind adjudication (SIMULATED)
Round 3Juice Shop held-out 3/9, 6 false (SIMULATED)
Fresh real confirmation runs1 verified path of 18 candidates; 0/14 multi-finding links (VERIFIED)
Passive engine on real taps18 to 27 direct exploitations per lab run, 0 chains (SIMULATED)
Active replay on crAPI2 direct exploitations, 0 multi-finding chains (SIMULATED)

Root cause. Interventions are the right idea causally, but they need lab-side references, authorisation for active traffic and a target that cooperates. Precision was good where they fired; recall was too low, and the rare errors were "proven" errors.

5. "A step is proven by observation alone" (November 17)

To get past zero, the rule was relaxed: corroboration by an endpoint-bound observed flow, without demonstrated necessity. Campaign c5 went from "0 → 2 proven". A baseline audit then showed both links were bound to their command only by the declared URL, and finding A had been pushed about 17 minutes later by another agent. Campaign c6's three "proven" paths rested on a wrong binding: a pg_hba finding bound to a refused cat /etc/shadow (VERIFIED). Under the later architectures, all of them fall to 0.

Root cause. Relaxing the definition raises the count, not the truth. The same audit found the proof path had 4,553 executable lines with 73 rules, 20 of them server-side guessing, and that 14 of 39 rules could be disabled without failing a test.

6. Execution ids cited by the agent (architecture C, November 17)

Every command output showed an opaque execution id; agents cited them on push; the server verified each edge on its own records. Technically it held: 0 false proofs on 66 critical negatives and a 95.8 % mutation score on the sealed harness (EXP-044, SIMULATED). Recomputed on real c5/c6 data it gave 0 proven, because old data had no ids. And the token add-on was +1.02 % and +1.75 % of a campaign, above the +1 % cap (EXP-046, SIMULATED).

Root cause. A correct design can still fail on cost. The verifier code stayed (dormant); the agent surface was removed, which led to architecture D.

A footnote that became a rule

On 11 November a capstone report on a three-host lab claimed "2 proven cross-host chains". They were chains the agent declared at finalisation, rendered proven because every step was independently exploited; the report itself noted that per-step necessity was not shown. On 12 and 17 November the product rules changed so that a declared chain is never proven. The claim is recorded as SUPERSEDED, not deleted.

Limits

  • Several earlier figures exist only in internal notes, not in primary logs; the repository labels them OBSERVED or UNKNOWN.
  • Source inconsistencies are recorded, not resolved: for example, the c6 recompute under architecture C is "1 → 0 proven" in one report and "3 → 0" in another.
  • "Failed" means the experiment missed its own criterion or was rejected. It does not mean the approach can never work elsewhere; counterfactual replay, for instance, was accurate where it applied.
  • The architecture that replaced these, D, has its own unsolved limit: 0 server-qualified links on real data.

Figures

Diagram. Figure 1. How each failure fed the next design. Simplified from the repository's architecture history.
Figure 1. How each failure fed the next design. Simplified from the repository's architecture history.
Figure 2. The interface consequence of these failures: an "Observed" group became "Possible / Inferred / Likely" groups with dashed lines. The demo seed also differs between the two captures (see the screenshot metadata).

Scientific sources

Research, source code & reproducibility


If you are about to build attack paths on top of automated findings, the failed-experiments folder is probably the cheapest hour you will spend on the project.

Next
Run it against your own lab

Darkmoon is open source (AGPL-3.0) and self hosted. Clone it, point it at a target you own, and read every line.