Why proof of exploitation beats AI vulnerability scores

Most AI vulnerability scanner output arrives as a number. A severity, a confidence percentage, a risk score, sometimes a tidy 0 to 100 with a color attached. It reads as certainty, but it is a prediction. The tool looked at a response, matched a pattern, and estimated the odds that the thing it saw is exploitable. That estimate is useful for triage volume. It is not the same as knowing the finding is real, and buyers evaluating AI security tooling are increasingly being asked to pay premium prices for the estimate.

Darkmoon takes the opposite stance. Every one of the 50 specialist agents carries the same status rule, and the rule is built around evidence, not likelihood. A finding is only allowed to claim impact if the agent can show the exact request or payload it sent, the raw response it got back, and the data it extracted or the effect it produced. No proof, no impact claim. This article is about why that distinction matters more than any score, and how it changes what a security leader actually does with a report.

A score is a guess, a proof is a fact

A CVSS base score answers a hypothetical: if this vulnerability exists as described and if it is reachable in your environment, how bad would it be. It is a severity model, and a good one. What it deliberately does not tell you is whether the specific instance in front of you is genuinely exploitable right now, on this host, with this configuration. That second question is the one that costs your team time, and it is the one a probability score papers over.

Proof of exploitation collapses the hypotheticals. Instead of asserting that an endpoint looks like it is injectable, the agent sends a payload, records the server behavior, and shows you the extracted row, the executed command output, or the token it read. The finding is no longer a claim about what is probable. It is a captured event you can reproduce. A report full of proofs is a report you can act on without re-testing every line, which is the entire economic argument for autonomous testing in the first place.

The false-positive tax of probability-based scanners

Every false positive has a cost, and it is paid by your most expensive people. A scanner flags a suspected SQL injection at 82 percent confidence. Someone senior now has to open the finding, reconstruct the request, try to reproduce it, decide it was a reflected error message and not injection, write that down, and close it. Multiply that by a queue of hundreds and the tool has quietly converted its own uncertainty into your payroll. The score did not save the work. It deferred it to a human and relabeled it triage.

Pattern-matching scanners are structurally prone to this because they reason about surface signals. A 200 OK on a path that should not exist, a response that differs when a quote is added, a payload reflected back in the page, a config value that mentions a secret. Each of those is a lead, and each is routinely reported as a finding. The gap between those two words is the false-positive tax. Proof-first testing refuses to cross that gap on a signal alone. It tries to turn the lead into an executed effect, and when it cannot, it says so plainly instead of hiding behind a confidence figure.

The status model: EXPLOITED, CONFIRMED, UNCONFIRMED

Darkmoon does not emit a single confidence number. It assigns one of three statuses to every finding, and each status has an explicit evidence bar the agent has to clear before it is allowed to use it.

The important design choice is the ceiling on UNCONFIRMED. A lead the agent could not turn into an effect is not allowed to masquerade as a high-severity item. It is capped at low severity and a CVSS of 3.9 or below, so it can never dominate a queue that also contains proven compromise. This is the structural fix for the false-positive tax: the uncertain finding is still reported, because hiding it would be dishonest, but it can never outrank a finding that ships with a raw response behind it.

Adversarial self-challenge before any CONFIRMED claim

The status alone would not be trustworthy if the agent were free to grade its own homework generously. So before a finding is allowed to keep a CONFIRMED or EXPLOITED label, the agent runs an adversarial self-challenge against its own evidence. It asks, in effect, what would make this claim wrong, and it demotes the finding whenever the honest answer is that the evidence does not actually show impact. The demotion rules are specific and they map directly onto the classic sources of scanner false positives:

• A bare 200 response on a route, with no proof the route does anything sensitive, is a lead, not a finding.
• A differential response, where the server behaves differently for a probe, is demoted unless the difference was turned into an actual extraction or action.
• An echoed payload reflected back in the page is not proof of execution, and does not by itself justify an injection claim.
• A file served but not executed is exposure, not code execution, and it is graded as exposure.
• A secret that is public by design, a published key meant to be published, is not reported as a leak.

This is not a slogan. It shows up in the published campaigns. In our unauthenticated Redis run, the agent tried the classic RDB-write path to remote code execution, hit the Redis 7.x protected-config error, recorded the exact refusal, and demoted its own RCE claim to mitigated. The campaign closed with data-layer compromise and denial of service, a smaller and accurate headline, rather than the bigger claim that would have failed the first time a customer tried to reproduce it.

CONFIG SET dir /tmp
-> (error) ERR CONFIG SET failed (possibly related to argument 'dir')
   - can't set protected config

The same discipline produces reports with no exploitation at all when that is the truth. Our public EBS snapshot assessment ran against a live AWS account with a deliberately limited identity and exploited nothing, and the report says so. The hardened website case is the clearest example: sixteen findings, zero critical, zero high, nothing exploited.

A probability scanner has no incentive to publish that report. A proof-first one has no other option, because there is nothing to show. The full write-up is our hardened site assessment, and it is the most useful boring report we shipped.

How this maps to CVSS, MITRE ATT&CK and bug bounty triage

Proof does not replace the frameworks security teams already use. It feeds them better inputs. The FIRST CVSS v4.0 specification separates the base metrics from the threat metrics precisely so that real-world exploitability can adjust the score. An EXPLOITED finding is exactly the evidence that justifies raising exploit-maturity from the theoretical to the observed, and an UNCONFIRMED lead is exactly what should not carry a threat uplift. The status model is a clean source for that adjustment instead of a guess about it.

For MITRE ATT&CK, a proof maps a finding to a technique that was actually performed, not one that was inferred. When the agent extracts a token and uses it, that is a concrete execution or credential access step you can place on the matrix and defend against, rather than a suspicion that a technique might apply. Detection engineering benefits from the exact request and response far more than from a severity label.

Bug bounty triage is the purest expression of this culture. HackerOne and Bugcrowd triagers reward reports with a working proof of concept and downgrade or close reports that assert impact without demonstrating it, and Bugcrowd's Vulnerability Rating Taxonomy exists to keep severity honest. A Darkmoon CONFIRMED finding already carries what a triager asks for: the exact payload, the raw response, and the extracted data. The status model is, in effect, the triage bar applied before the report is even written.

Why proof changes remediation priorities

A queue sorted by predicted severity and a queue sorted by proven impact are different queues, and they send your engineers to different work. When findings carry proof, the team fixes what is demonstrably exploitable first, because those items have a captured effect attached and cannot be argued away in a meeting. The UNCONFIRMED leads still get logged, but they sit where they belong, below the proven items, capped at low severity so they never crowd out real compromise.

Proof also shortens the argument between security and engineering. A developer can dispute a probability. It is much harder to dispute the raw response that shows their session token being read by an unauthenticated client. The remediation conversation moves from whether the finding is real to how fast it gets fixed, which is where a security leader wants it. If you are evaluating an AI vulnerability scanner, the questions worth asking are direct:

• For each high-severity finding, show me the exact request and the raw response. If the tool cannot, the severity is a prediction.
• Show me a finding this tool demoted after failing to reproduce impact. A tool that never downgrades itself is not testing, it is guessing confidently.
• Show me a clean report, one with no critical or high findings. A vendor that cannot produce a boring report is optimizing for alarm, not accuracy.
• Ask where the reasoning ran and what left your network. Darkmoon runs on a local model and its agents operate through a controlled MCP layer, so the reasoning stays on your infrastructure.

What we deliberately refuse to claim

FAQ

What is proof of exploitation? It is evidence that a vulnerability was actually exercised, not just detected: the exact request or payload sent, the raw response received, and the data extracted or the effect produced. It converts a claim about what is probable into a captured event you can reproduce.

Are AI vulnerability scores reliable? They are a reasonable way to rank a large volume of leads, but a score is a prediction about exploitability, not a demonstration of it. Treated as a triage aid it is useful. Treated as a verdict it produces false positives that your team pays for in re-testing time.

How does proof reduce false positives? By refusing to promote a lead to a finding without an executed effect behind it, and by capping anything unproven at low severity. Darkmoon demotes bare 200 responses, differential responses, echoed payloads, files that were served but not executed, and secrets that are public by design.

Does a proof replace a CVSS score? No, it improves it. Proof is the real-world evidence the CVSS v4.0 threat metrics are designed to consume. An EXPLOITED finding justifies an exploit-maturity uplift, and an UNCONFIRMED lead is exactly what should not receive one.

How do bug bounty programs treat proof of exploitation? Platforms like HackerOne and Bugcrowd reward reports with a working proof of concept and downgrade or close reports that assert impact without demonstrating it. A Darkmoon CONFIRMED finding already ships the payload, raw response and extracted data a triager expects.

What proof-first testing actually buys you

The value of autonomous security testing is not the count of findings. It is the confidence that each one survives contact with reality. A tool that hands you a hundred scored maybes has moved its uncertainty onto your team. A tool that hands you the proofs, the honest demotions, and the occasional clean report has done the triage work for you and shown you its receipts. That is the difference between a score and a fact, and for a security leader deciding where the week goes, it is the whole game.

Darkmoon is GPL-3.0 and self hosted, and every finding it publishes carries its proof. Read the reports, then run it against something you own: source, docs.