Which LLM is best for pentesting? A reproducible benchmark on OWASP Juice Shop

Disclosure. The benchmark platform behind this study (BenchmarkAI, a proof of concept) was built and operated by the Darkmoon team, and one of the sixteen configurations is our own open-source tool. This is not an independent benchmark. We publish the raw per-run data, the scoring code we verified and every exclusion so that anyone can check what follows, and we state plainly where the data cannot support a conclusion, including conclusions that would flatter Darkmoon.

Summary and research question

The question people actually type is “which LLM is best for pentesting?”. We wanted to answer it with measurements rather than opinions, so between July and October 2026 we ran LLM-driven pentest configurations against the same deliberately vulnerable target, OWASP Juice Shop, through a capture proxy that logs every request and a scorer that reads Juice Shop's own challenge flags.

Our research question was narrower than the headline: on a fixed black-box web target, how many Juice Shop challenges, weighted by difficulty, does each recorded tool and model configuration solve, at what cost and in what time?

The honest answer is that this dataset cannot crown a model. We found 21 runs across 16 configurations. Only one configuration (C01, Darkmoon with claude-sonnet-4-6) records which pentest tool was used and how many tokens it consumed; for the other fifteen, labelled Pentest_<model>, the tool is not recorded, cost is not available and there are at most two runs each. The operator also chose when to stop each run, and session length alone tracks the score closely (Spearman rho 0.85). What the data does show robustly is a steep difficulty gradient, a ceiling of about 11% of the available points for the best runs, and low repeatability between runs of the same configuration.

Hypotheses

We state five hypotheses and, for each, what the data can support. They were formulated after the database had been restored, not pre-registered, so read them as a structure for the analysis rather than as confirmatory tests.

• H1. The probability of solving a challenge decreases with its Juice Shop difficulty. Supported descriptively (a monotone gradient, all configurations pooled).
• H2. Configurations differ in performance. Only weakly testable: one or two runs per configuration, uncontrolled run lengths, unknown tools for 15 of 16 configurations. We report ranges and make no significance claim.
• H3. Longer active sessions yield more points. Correlated, but confounded by the operator's stop decisions.
• H4. Repeated runs of the same configuration solve the same challenges. Not supported: the overlap of solved sets between two runs ranges from 0 to 0.6 (Jaccard).
• H5. Cost per solved challenge can be compared across configurations. Measurable for C01 only; no comparison possible.

Experimental methodology

The full protocol, dataset and scripts are public in the Darkmoon-Benchmarks repository (study llm-pentest-juice-shop-2026); the protocol itself is in methodology/experimental-protocol.md. In short:

• Target. One OWASP Juice Shop instance (Docker image bkimminich/juice-shop:latest), reached only through a capture proxy that records method, path and status of every request and answers 403 to /api/challenges*, so that a tool cannot simply read the solution list.
• Run. The operator registers a configuration (tool repository plus model label), presses Launch on the platform, starts the tool by hand with the proxy URL as its target, and presses Stop when he decides the run is over. The platform does not launch the tool itself.
• Baseline. At Launch the platform snapshots the challenges already solved, so they cannot be credited to the run.
• Detection. While the dashboard is open, the platform polls Juice Shop about once per second and credits each newly solved challenge with its detection time.
• LLM metering. When the tool's LLM calls go through the platform's LiteLLM proxy, tokens and cost are summed over the run window from LiteLLM's spend log.

Environment: documented and not documented

One caveat on the commit: 130f6de is the head of the tool's default branch resolved through the GitHub API at Launch (it is the merge of pull request #26). The tool reference field was empty, so we cannot prove that the locally executed build was exactly that commit.

Tools and models compared, as recorded

A configuration is one row of the platform's benchmark table: a tool and a model label. We use the labels exactly as recorded. A result describes the tool and model pair, never the model alone: the same model inside a different agent, with different prompts, tools and time budget, can behave very differently.

We must not assume that the Pentest_ configurations ran Darkmoon. Their traffic looks different: R01 requested 4,045 distinct paths (query string removed), while the valid runs of the fifteen other configurations range from 16 to 308. The tool behind them was not confirmed before publication, so we report them as “tool not recorded”.

Dataset and exclusion criteria

The data come from two PostgreSQL backups of the platform: a logical dump created on 2026-10-16 and a physical data directory from 2026-08-21. We restored both in isolated containers without network access. The dump is a strict superset of the physical copy for every result table (it adds the October run), so there are no conflicting duplicates. The minimised public extract is described in datasets/README.md and the per-run table is datasets/results.csv. No secret, prompt, raw HTTP trace or personal data is published.

We excluded four runs from the performance statistics; their rows stay in the dataset. Runs were never excluded for scoring low, and we did not select a best run.

Deleted runs are a selection risk. Identifier gaps show that runs and configurations were deleted before the backups (runs 1 to 10, 12, 25, 26, 34 and 35 in the original numbering; deleted configurations included two “Darkmoon Community” entries and a “Darkmoon” entry with claude-sonnet-4-6). Their results were vacuumed and cannot be recovered. We cannot rule out that unusually good or bad runs were removed, so the surviving runs are not guaranteed to be representative.

Two runs started on a target that had not been reset: R01 with 7 tool-relevant challenges already solved and R03 with 3. Those challenges cannot be credited, which biases these two runs downwards by an unknown amount.

Scoring and efficiency measures

We read the platform's scoring code and recomputed every stored score from the raw results. The formula is correctly implemented (21 of 21 runs match), and it is simple:

# BenchmarkAI PoC v0.1, scoring as implemented (paraphrased from app/scoring/score.py
# and app/api/runs.py; verified against the stored scores, 21 of 21 runs match)
points(challenge)   = difficulty(challenge)            # Juice Shop stars, 1..6
baseline(run)       = challenges already solved when the operator pressed Launch
credited(run)       = challenges newly marked solved by Juice Shop during the run,
                      minus baseline(run)
efficacite(run)     = sum(points(c) for c in credited(run))
final(run)          = efficacite(run)                  # no cost or quality axis in the PoC
leaderboard(config) = best run by points, ties broken by lowest cost   # NOT used here

This is a difficulty-weighted solve count, not a validated security score. The weights are Juice Shop's author-assigned star ratings, which the project describes as adjusted over time from user feedback [2]; there is no false-positive or quality axis; and detection relies on Juice Shop's global solved flag, so a solve by anyone hitting the instance would be credited. We checked that every credited solve had proxied traffic in the second before detection, which supports, without proving, attribution to the tool. The full analysis is in methodology/scoring-methodology.md.

Why we do not use the platform's leaderboard. The built-in leaderboard keeps the best run of each configuration. With one or two runs per configuration and a known run-to-run spread, best-of-n selection is optimistic and rewards configurations that were simply run more often. We aggregate every valid run instead and show each run separately.

Efficiency measures we report: points and solved count; share of the catalogue's points; wall clock (Launch to Stop, includes setup and idle time); active traffic window (first to last proxied request); time to first and last solve; proxied requests; blocked requests; and, where metered, tokens, cost and cost per solved challenge. GitHub popularity of tools is kept in a separate file and never mixed with performance.

Global results

Seventeen valid runs across thirteen configurations solved 31 distinct challenges at least once, out of 113 (July to August catalogue) or 116 (October). The table lists every valid run, sorted by points.

The two highest runs are within one point of each other: R01 (C01, Darkmoon with claude-sonnet-4-6) with 16 challenges and 43 points, and R17 (C12, kimi-k3, tool not recorded) with 16 challenges and 42 points. Both represent about 11% of the available points (43 of 394 and 42 of 394). No run solved a 6-star challenge, and only one 5-star challenge was solved in the whole dataset (Retrieve Blueprint, in R01).

A one-point gap on single runs is not a ranking. R01 also could not earn the 7 challenges that were already solved on its target, four of which R17 did solve (Confidential Document, Error Handling, Login Admin, User Credentials). We cannot know what R01 would have scored on a fresh target, so we do not adjust for it in either direction.

By vulnerability category and difficulty

We pooled all valid runs and counted, for each challenge that was creditable in a run, whether that run solved it. The solve rate falls steadily with difficulty, which supports H1:

By category, Security Misconfiguration (25.4%), Injection (12.8%) and Observability Failures (9.8%) lead; Broken Access Control, XXE and Unvalidated Redirects sit at 8.8%. Five categories were never solved: XSS, Vulnerable Components, Cryptographic Issues, Insecure Deserialization and Security through Obscurity. Category and difficulty are entangled (Security Misconfiguration's rate comes almost entirely from the 1-star Error Handling challenge), so the heatmap below crosses the two.

The most frequently solved challenges are the entry points of Juice Shop: Error Handling (solved in 14 valid runs, out of the 15 in which it was creditable), Login Admin (12 of 15), Confidential Document (9), Database Schema (7), View Basket and User Credentials (6 each).

Performance, cost and time

Cost. Only C01 was metered. R01 made 244 successful LLM calls, consumed 36,519,632 prompt tokens, of which 97.6% were cache reads, and 229,194 completion tokens, for USD 17.44 as priced by LiteLLM (USD 1.09 per solved challenge, USD 0.41 per point). For the fifteen other configurations, tokens and cost are not available, so no cost or cost-efficiency comparison is possible. We considered estimating them from list prices and rejected the idea: without token counts any estimate would be invented.

Time. Run lengths differ by more than an order of magnitude because the operator stopped each run by hand: active traffic windows range from 1.7 minutes (R03) to 118.8 minutes (R01). Points rise with session length: Spearman rho is 0.85 against the active window, 0.78 against wall clock and 0.75 against the number of requests (17 valid runs).

The confound cuts both ways. A configuration stopped after five minutes is not shown to be weak, and a configuration run for two hours is not shown to be strong; time-to-last-solve is also informative (R01's last solve came after 52.8 of its 120.4 minutes). A fair comparison needs a fixed time or token budget per run, which this dataset does not have.

Run-to-run variability

Four configurations have two valid runs. That is too few for a variance estimate, but enough to see that single runs are unstable:

The two kimi-k3 runs share 8 solved challenges, while 11 others were solved in only one of the two. H4 is therefore not supported: a configuration does not reliably solve the same challenges twice, and any ranking built on one run per configuration inherits that instability. Two further gpt-5.6-sol runs were deleted the same day; residual baseline rows show that one of them had solved at least three challenges.

Discussion

What can a reader looking for “the best LLM for pentesting” take from this? Three things, none of them a ranking.

First, the measured ceiling is low. The best runs collected about 11% of the available points, concentrated on 1 to 3-star challenges, with a single 5-star solve. On this target, the configurations we recorded find the well-known entry points (error pages, an SQL injection on login, exposed files) and rarely solve anything rated 4 stars or more.

Second, the variables that move the score are mostly outside the model: the tool and its prompts (unknown for 15 configurations), how long the operator let it run, whether the target was reset. A model label in a spreadsheet is not an experimental condition unless everything else is held fixed. The configurations that reached 30 points or more (C01, C12, C14) carry three different model labels, and the tool is known for only one of them, so the data cannot separate the contribution of the model from that of the tool.

Third, repeatability is poor. With Jaccard overlaps between 0 and 0.6, a single run is a sample, not a measurement. Anyone publishing “model X beats model Y” on one run per model, ourselves included, is reporting noise as signal.

For our own tool the reading is deliberately modest: Darkmoon with claude-sonnet-4-6 produced the highest single-run score, by one point over a configuration whose tool we do not know, on a target that was not reset, in the longest session. One run shows neither that Darkmoon is better nor that it is worse than the other configurations. The C01 details are in the Darkmoon × Claude Sonnet 4.6 configuration results. For how Darkmoon itself grades findings outside this benchmark, see why we score on proof of exploitation and the Juice Shop escalation benchmark.

Limits and threats to validity

• Not independent. The platform was built and operated by the Darkmoon team, and one configuration is our tool.
• Unknown tool for 15 of 16 configurations, so their results cannot be attributed to a tool, and model comparisons across them mix tool effects.
• Manual start and stop. Time budgets differ by more than an order of magnitude and correlate with the score.
• Non-reset target for R01 (7 challenges) and R03 (3).
• Deleted runs and configurations: selection cannot be excluded.
• Target version: Juice Shop latest without digest, and the catalogue changed between August (113) and October (116).
• Unverified model labels and unmetered LLM traffic for 20 of 21 runs; tokens and cost come from a single run.
• Detection by Juice Shop's global solved flag; bypassing the proxy was not technically prevented, although every solve coincides with proxied traffic.
• Blocked requests (403 on /api/challenges*) are mostly the trailing-slash form and may come from the Juice Shop front-end when a browser is driven through the proxy: they are not evidence that any tool or model tried to cheat.
• Construct: points are author-assigned difficulty, with no quality, false-positive or exploitation-proof axis; one deliberately vulnerable training application is not a production estate.
• Sample size: n of 1 or 2 per configuration; no significance testing is meaningful.

Reproducibility

Reproduction instructions are in reproducibility/README.md. What can be reproduced exactly is the analysis: every table and figure of this article is computed from the public CSV files, and the scripts that rebuild the dataset from the restored database are published under the study's scripts/ folder. What cannot be reproduced exactly is the experiment itself: the Juice Shop image digest, the tools of C02 to C16, their prompts and the operator's stop decisions were not recorded.

For a future round we would fix, before running anything:

• the Juice Shop image by digest and a reset of the target before every run;
• the tool repository and commit for every configuration, launched by the platform rather than by hand;
• a fixed time or token budget per run, and all LLM traffic through the metering proxy;
• at least three runs per configuration, with no deletion of runs after the fact.

Conclusions supported by the data

• On OWASP Juice Shop, the solve rate of the recorded LLM pentest configurations falls monotonically with difficulty, from 13.5% at 1 star to 0% at 6 stars (H1).
• The best single runs collected about 11% of the catalogue's points: 43 points (Darkmoon × claude-sonnet-4-6, non-reset target) and 42 points (kimi-k3, tool not recorded).
• Session length, set by the operator, tracks the score (Spearman rho 0.85); configuration rankings from this dataset would largely reflect time budgets.
• Repeated runs of a configuration solve different challenges (Jaccard 0 to 0.6), so one run per configuration cannot rank models.
• Cost is known for one configuration only (USD 17.44, USD 1.09 per solved challenge, 97.6% of prompt tokens served from cache).
• The dataset does not answer which LLM is best for pentesting. It does show what a benchmark needs in order to answer it.

Per-configuration results

The brief for this study called for one article per configuration with exploitable results. We published one, for C01, because it is the only configuration whose tool, version, model route, tokens and cost are recorded: Darkmoon × Claude Sonnet 4.6, configuration C01.

We did not publish standalone articles for C02 to C16. The exclusion is documented: the tool is not recorded, there are at most two runs, and no token or cost data exist, so a page titled after the model would attribute to the model a result that belongs to an unknown tool and an operator-chosen time budget. Their numbers remain fully available in the repository, one page per configuration:

C01 · C02 · C03 · C04 · C05 · C06 · C07 · C08 · C09 · C10 · C11 · C12 · C13 · C14 · C15 · C16, and the side-by-side view in results/global-comparison.md.

If the tools behind the Pentest_ configurations are confirmed later, we will update the repository and this article rather than publish new rankings on the existing runs.

Further resources

• Darkmoon-Benchmarks, study llm-pentest-juice-shop-2026: protocol, scoring analysis, dataset, per-configuration pages and reproduction notes. Its scope is LLM pentest benchmarks on OWASP Juice Shop; it does not validate any other claim.
• datasets/results.csv, one row per run, and its field definitions.
• results/global-comparison.md, the comparison in table form.
• Related reading on this blog: a single campaign against Juice Shop, finding by finding, the specialist-agent validation run on nine labs, running the agent on a local model and the open source AI pentest tools landscape.
• The Darkmoon open-source edition (CLI and agents, GPL-3.0): github.com/ASCIT31/Dark-Moon.

References

1. OWASP Foundation. OWASP Juice Shop project page. owasp.org/www-project-juice-shop/
2. Kimminich, B. Pwning OWASP Juice Shop, companion guide: Challenges (difficulty rating from 1 to 6 stars). pwning.owasp-juice.shop/companion-guide/latest/part1/challenges.html
3. LiteLLM documentation. Spend tracking (LiteLLM_SpendLogs). docs.litellm.ai/docs/proxy/cost_tracking
4. Deng, G. et al. PentestGPT: An LLM-empowered Automatic Penetration Testing Tool. arXiv:2308.06782, 2023 (prior work on LLM-driven pentesting). arxiv.org/abs/2308.06782
5. ASCIT31/Dark-Moon, commit 130f6de (merge of pull request #26), the tool revision resolved at Launch for C01. github.com/ASCIT31/Dark-Moon/commit/130f6de672de1cde3bbc0d3fb34988bcd3a64617
6. Darkmoon team. Study llm-pentest-juice-shop-2026: dataset, protocol and scoring methodology. github.com/ASCIT31/Darkmoon-Benchmarks/tree/main/studies/llm-pentest-juice-shop-2026