· 6 min read
The hardest blocker for AI driven penetration testing is not the reasoning. It is data exposure. The moment an autonomous agent sends a real IP, hostname, credential or internal path to a hosted model, sensitive facts about your infrastructure leave your perimeter and land in a third party's logs. For regulated, defence and sovereignty constrained teams, that single fact rules the whole approach out.
The usual answer is "just run a local model". It works, but it trades away most of the reasoning quality that made the approach interesting in the first place. We wanted both: frontier model reasoning, and secrets that never leave the building. This is the design we landed on.
The idea: the model only ever sees placeholders
We put an anonymization gateway between the agent and the model. Every sensitive value is replaced by a deterministic placeholder before anything reaches the LLM: IP_PRIVATE_001, HOST_INTERNAL_001, EMAIL_001.
Deterministic is the load bearing word. The same real value always maps to the same placeholder within a run, so the model can still reason about relationships (this host talks to that host, this credential works there, this path repeats across services) without ever seeing the real data. Strip determinism and the agent loses the ability to correlate, which is most of what makes it useful.
Rehydration happens locally, just in time
Placeholders are useless to a scanner. So the real values live in a local vault and are reinjected locally, just in time, right before a tool actually executes, then masked again before any tool output travels back to the model. The window in which a real value exists in clear is the tool invocation itself, on your own machine.
- Agent reasons on placeholders and decides what to run.
- The gateway rehydrates the real values locally, at the last possible moment.
- The tool runs against the real target.
- Output is masked back to placeholders before the model ever reads it.
A command gateway blocks exfiltration
Tokenisation alone is not enough: an agent can always try to be clever. A command gateway inspects what the agent attempts to run and blocks exfiltration patterns, for example curling secrets to an external host, or encoding and leaking the placeholder mapping itself. It is a policy layer, and we are careful to describe it as exactly that.
Does it actually still find things?
That is the fair question, because a privacy layer that halves detection is not a feature. We validated it end to end on OWASP Juice Shop with the gateway active for the entire run: 56 vulnerabilities found, and the model never saw a single real IP or email. Every finding ships with proof of exploitation, the executed payload and the observable effect, rather than a model asserting that something "looks like" SQL injection.
What this does not solve
We would rather state the limits than have you discover them:
- Deterministic placeholders leak structure by design. Cardinality and co occurrence remain visible to the model even when values are not. For most infrastructure data that is an acceptable trade, but it is a real information channel and we treat it as one.
- The command gateway is a policy layer, not a sandbox. It shrinks the exfiltration surface. It does not make a genuinely hostile model safe, which is why tools run in containers on top of it.
- Juice Shop is a deliberately vulnerable teaching app. The number proves the gateway does not blind the agent; it is not a claim about hard targets.
Try to break it
The gateway core is open source, like the rest of Darkmoon (GPL-3.0). If you work on agent security, we would genuinely rather you break the placeholder scheme, attack the rehydration timing, or find an exfiltration path the command gateway misses, than take our word for any of this. The documentation covers the threat model in detail.