Remediation benchmark

Automated vulnerability remediation, proven end to end

Most autonomous pentesters stop at the finding. Darkmoon's Pro remediation engine turns each one into an AI remediation pull request — and we published the honest score: 42 of 57 findings fixed and retested end-to-end on OWASP Juice Shop.

Read the full dossier See the 57 pull requests

findings, each mapped to its own addressable PR (#1–#57)

demonstrated end-to-end with a clean live exploit-retest

distinct fixes across the 57 PRs (deduplication disclosed)

pull requests merged — every one awaits human review

The loop

Finding, fix, retest, review

The same four steps run on every finding. A fix only counts toward the 42 when the original exploit is re-run against a live instance and confirmed closed.

Finding

A vulnerability the autonomous pentest already proved with a working exploit is mapped back to the exact source file(s) that govern it.

Fix

The remediation agent generates a minimal, root-cause patch against the real Juice Shop code — not a suppression or a config toggle.

Compile + retest

The branch is type-checked with tsc, then the original exploit is re-run against a live instance. The finding must be confirmed closed.

Human-reviewed PR

A pull request is opened for a human to review. It is AI-generated code, disclosed as such, and never auto-merged.

Explore the same loop visually, one finding walked from exploit to green retest, on the interactive walkthrough.

Honest limits

The 15 we did not count

42 is the number that survives a clean runtime retest. The other 15 are disclosed here rather than folded into a bigger headline.

Count	Category	Why it is excluded from the 42
9	Partial draft PRs	Close a narrower exploit at runtime but are partial or architectural relative to the full finding.
1	PR #52 — ineffective fix	Did not close the exploit on retest. The fix is ineffective and is flagged honestly.
1	PR #3 — not runtime-retested	Verbose errors: changes behaviour only under NODE_ENV=production, so it was not runtime-retested.
4	Frontend / asset PRs	#44, #46, #18/#39 — frontend or static-asset changes not exercised by the API-level retest harness.
15	Total excluded	57 findings − 42 demonstrated = 15 disclosed above.

Method

How each PR was checked

Maps to a real, exploit-proven finding in the pentest dataset.
The diff targets the real Juice Shop file(s) governing the finding — no suppression.
Type-checks with tsc (24 distinct backend branches, all exit 0).
The exploit is reproduced on main then re-run against the fix branch on a live v19.2.1 instance.
Scanned for secrets; PR bodies disclose the fix is AI-generated and awaits human review.

Reproduce it

Everything is in the open

The target is a standalone MIT copy of OWASP Juice Shop v19.2.1. The retest harness, the per-PR verdict matrix and the exact reproduction environment are all published. These PRs are an internal demonstration of the remediation engine on our own fork — not upstream contributions.

REMEDIATION_BENCHMARK.md (full 57-PR matrix) The 57 pull requests on ASCIT31/juice-shop The open offensive benchmark (57 findings, black-box)

This is Darkmoon's own reproducible benchmark on a public lab. It is not a third-party certification or an analyst endorsement. Every number above is drawn from the published dossier and can be reproduced or contested.

Test the loop on your own code

Darkmoon is open source (GPL-3.0) and self hosted. The offensive engine is free; the remediation engine is a Pro capability.

View on GitHub Compare the tools