Benchmark: autonomous specialist agents against nine vulnerable labs, with proof and honest limits

When we expanded Darkmoon’s specialist agent roster, we did not ship the new agents on faith. We ran an end-to-end validation: each new agent had to be correctly dispatched by the orchestrator on a real artifact, run its playbook against a live vulnerable lab, push findings with full evidence, and have the campaign finalize to a report. This article is that benchmark, with the real numbers, what was proven, what we deliberately held back, and what we could only validate at the command level.

Coverage, at a glance

Nine specialist agents were exercised against live labs. The finding counts are what each campaign actually recorded; where three agents ran as one parallel cascade against a shared scenario, the findings are reported as the combined total for that cascade rather than split after the fact.

What the agents actually proved

The point of the benchmark is not the counts, it is the impact behind them. Highlights, each one executed against the lab:

• messaging-cache / Redis. Unauthenticated access exploited; the seeded session:admin token extracted; CONFIG SET, EVAL and a SLAVEOF rogue-master demonstrated.
• sql-databases. PostgreSQL COPY … TO PROGRAM command execution; pg_read_file reading /etc/passwd; password-hash extraction from pg_shadow and mysql.user; and the seeded api_keys and PII rows exfiltrated, using the real psql and mysql clients.
• hashicorp-vault. Root-token compromise; KV v2 secrets extracted (a seeded AWS key and a database password); an orphan root token minted to demonstrate persistence.
• container-registry. Anonymous catalog listing; a hardcoded AWS key and PostgreSQL credentials pulled from a seeded image layer; an anonymous-push supply-chain substitution.
• docker. Unauthenticated Engine API to host root; a container escape reading the host /etc/shadow; real container-environment secrets and a mounted kubeconfig discovered. A genuine end-to-end host compromise via the socket.

What we deliberately did not claim

The most important result in any honest benchmark is the finding you refuse to inflate. On Redis, the classic RDB-write remote code execution was demoted to mitigated, not reported as exploited, because Redis 7.x protects against it. The agent tried it, it did not work on this version, and the report says so. That is the status discipline working: a benchmark that only ever counts up is a benchmark you cannot trust.

The limits: what could only be command-validated

Some agents target platforms that have no faithful local emulator, so they could not be driven to exploitation against a lab. We validated those at the level the environment allowed and we label them exactly that way, rather than pretending to a live-exploitation result we did not get:

• github and gitlab were command-validated: there is no local API-faithful emulator, so the agents’ curl and jq flows were validated against the real APIs rather than exploited against a lab.
• azure, gcp and entra-id were preflight-validated: no local emulator exists, so the az and gcloud preflight and STOP-logic were confirmed rather than run to impact.

The bugs the benchmark surfaced

Running a benchmark honestly means publishing what broke. Three issues surfaced, and they were product issues, not agent failures:

• A stale-campaign reconciler flipped a long-running campaign to stopped before finalize completed, so the server report for the 22-finding SQL campaign returned a 500. The 22 findings were complete; the report was reassembled from them with the same schema.
• The report generator was not resilient to a non-numeric CVSS score and could 500 on a value like "9.8 (High)". It was hardened with a safe numeric parse across all repos.
• Sub-agent to parent finding aggregation remained partial: findings can occasionally cross-attribute between campaigns. Known, and called out.

There was also a test-environment caveat worth stating: an early Redis campaign hung because the running toolbox image carried a stale compiled allow-list that reported the Redis client as unavailable, pushing the model toward a raw socket that blocked forever. A clean CI rebuild of the toolbox image resolved it, and the rebuilt image was verified at 118 of 118 tools.

Why this is a fair test

# The benchmark contract, per agent:
1. orchestrator dispatches the agent on a POSITIVE artifact (not by guessing)
2. the agent runs its real playbook against a live lab
3. it pushes findings with full evidence (command, request, response)
4. the campaign finalizes to a server-generated report
5. status is graded by demonstrated impact, and over-claims are demoted

Every step is observable and reproducible: the labs are standard components, the dispatch is on a real artifact rather than a coin flip, and the evidence for each finding is recorded. You can stand up the same labs and run the same agents. For a single-target deep dive with the full request and response transcripts, the individual case studies cover Redis, PostgreSQL and MySQL, Vault, the registry and the Docker socket, the Terraform state and Jenkins.

Darkmoon is our open source project (GPL-3.0): github.com/ASCIT31/Dark-Moon, docs.