How the Darkmoon LLM agent pentests an AI inference endpoint, with garak

A model server is now part of the attack surface. Teams stand up an OpenAI compatible gateway, an Ollama instance, a vLLM or a text generation inference container, wire it into a product, and it quietly starts answering on the network like any other API. The Darkmoon llm specialist, shipped in release v1.4.0, treats that endpoint the way the rest of the roster treats a database or an identity provider: it detects it during discovery, dispatches a dedicated agent, and works the OWASP Top 10 for LLM Applications end to end with proof for every claim. This article is a walk through of how that agent actually runs, taken from the agent definition and a full end to end validation run, not from a marketing outline.

Detection and dispatch: the endpoint has to name itself

Darkmoon never dispatches a specialist on a guess. The orchestrator in pentest.md now runs a mandatory LLM probe during discovery, because an exposed model API is a positive network artifact that has to be actively looked for rather than inferred from the absence of other markers. On any responsive HTTP service it issues four bounded requests: GET /v1/models, GET /api/tags, a minimal POST /v1/chat/completions and a POST /api/generate, each carried under a curl --max-time.

A dispatch fires only on a concrete positive signal. In our validation run the orchestrator saw a clean OpenAI compatible fingerprint and logged the service before it dispatched anything:

GET /v1/models
{"object": "list", "data": [{"id": "vuln-gpt", "object": "model", "owned_by": "corp"}]}

# orchestrator records the artifact, then dispatches
svc-api  "OpenAI-compatible API (port 9100)"  ->  DISPATCH: llm

The full signal list the orchestrator accepts is deliberately concrete: a /v1/models body shaped as {"object":"list","data":[{"id":...}]}, an Ollama model list on /api/tags, a /v1/chat/completions response that returns either {"choices":[{"message":...}]} or a structured {"error":{"type":...}} on a 400 or 401, response headers in the openai-* or x-ratelimit-* families, or a server banner for vLLM, text generation inference, llama.cpp, LM Studio, Ollama or TGI. Any one of those is enough. The dispatch is automatic on a positive artifact, exactly like the GraphQL, Active Directory and Kubernetes agents, and unlike the credential gated cloud planes that only fire when the operator supplies a scope. When it hands off, the orchestrator passes the base URL, the API style, the model ids from the listing, and whether authentication was required.

Phase 0: fingerprint and capability profiling

The sub agent confirms the endpoint before it attacks it. It lists the models, sets its working model id to the first one returned, and establishes a baseline chat response so it knows the endpoint's normal behaviour: its refusal style, its headers, its finish_reason. That baseline is not busywork. Half of the later judgements depend on it, because a jailbreak is only a jailbreak if the same request that used to be refused now returns the disallowed content.

It then profiles the endpoint's capabilities, and the profile decides which attack modules are even worth running:

The agent re runs this profiling after any successful jailbreak, because a bypass can unlock capabilities that were not reachable from the default behaviour. That is codified as a multi cycle model: cycle one is the unauthenticated default behaviour, cycle two adds any system prompt or persona the endpoint accepts, and cycle three is the post jailbreak state where the agent re tests restricted operations against the newly loosened guardrail.

Phase 1: an optional, bounded garak pass

For systematic, machine readable coverage the agent can call garak, NVIDIA's open source LLM vulnerability scanner (Apache 2.0), which v1.4.0 adds to the toolbox and the tool allow list. It is pinned to 0.16.0 for reproducible builds and installed CPU only, because it scans a remote endpoint and needs no GPU of its own. The install is non blocking by design: if it fails, the image still builds and the agent falls back to its manual methodology.

The scanner is tightly fenced. It runs against the single discovered endpoint, with a focused probe set rather than the full catalogue, under a mandatory timeout, with low generations to respect the anti abuse rules, and it must emit a machine readable JSONL report. The agent is allowed at most two garak invocations per campaign.

export OPENAI_API_KEY="${LLM_KEY:-sk-none}"
export OPENAI_BASE_URL="${TARGET}/v1"
timeout 900 garak --model_type openai --model_name "$LLM_MODEL" \
  --probes promptinject,dan,encoding,leakreplay,xss,malwaregen,glitch,goodside \
  --generations 3 --report_prefix /tmp/garak_scan 2>&1 | tail -40

# then parse the JSONL for failed (vulnerable) attempts and turn each into a lead
jq -c 'select(.entry_type=="attempt" and .status==2)
  | {probe:.probe_classname, detector:(.detector_results|keys),
     prompt:(.prompt|tostring|.[0:200]), output:(.outputs[0]|tostring|.[0:400])}' \
  /tmp/garak_scan.report.jsonl | head -60

For an Ollama or raw HTTP endpoint the agent uses garak's REST generator with a config that maps the input and response fields, or falls back entirely to the manual methodology, which needs no scanner at all. The decision logic is explicit: if garak reports a hit, reproduce it manually with curl to confirm before pushing anything; if it returns nothing, mark the scanner done and continue manually; if it errors or returns empty twice, mark it failed with proof and stop, never blocking the campaign on it.

A candid note from our own run, covered in the case study, is that on the OpenAI compatible target the agent judged its adaptive manual attacks more productive than a generic probe sweep and did not spend a garak pass. We verified separately that garak is genuinely functional in the sealed toolbox: pointed at a mock through OPENAI_BASE_URL it exited 0 and produced its JSONL report, a hit log and an HTML report in under a second. It is a systematic complement that is available, not dead weight, and not a crutch the agent depends on.

Phase 2: adaptive manual attacks, each with a detector

This is the core of the methodology, and its distinguishing feature is that the agent is both the attacker and the judge. It drives the endpoint with curl and scores the responses with python3 and jq, and every attack chains to an explicit detector so a claim is scored rather than assumed. It seeds each run with a unique canary token, of the form DMK-<random>, so a leak or an obeyed injection is unambiguous rather than a coincidence.

• Direct prompt injection (LLM01). Instruct the model to ignore its instructions and emit a fixed marker. The detector is simple: the exact marker appears in the output.
• Indirect prompt injection (LLM01). Embed the injected instruction inside data the model is asked to summarise or translate. The detector is that the model obeys the embedded instruction rather than treating it as data.
• System prompt leak (LLM07). Ask the model to reveal its system or developer message verbatim, and through obfuscation if it refuses. The detector is hidden context text or a planted canary coming back.
• Sensitive information and training data disclosure (LLM02, LLM06). Probe for secrets and PII in context and for memorised secrets. The detector is a PII or secret regex match in the output.
• Jailbreak and guardrail bypass (LLM09). DAN, roleplay, encoding and hypothetical framing. The detector is adversarial: the refusal string has to be absent and the disallowed content actually present, compared against the baseline refusal, with no credit for a partial or hedged answer.
• Insecure output handling (LLM05). Ask the model to output a <script> tag, a javascript: or data: URL, an SSRF URL or a SQL statement verbatim. The detector is the executable sink in the raw output, and the severity depends on whether the output is rendered downstream.
• Excessive agency and tool abuse (LLM08), only when the profile found tools or functions: try to make the model call a tool with attacker controlled arguments. The detector is a tool call carrying those injected arguments.
• Bounded resource exhaustion (LLM04, LLM10). A handful of requests, never more, one with a very large token budget and one asking for unbounded output, measuring latency and whether an output cap exists. The finding is the absence of a cap, and a few requests prove it.

When a guardrail is present the agent profiles it before it tries to beat it. It records whether a moderation layer sits on the input, the output or both, and whether a block is a status code, a canned refusal string or a content_filter finish reason. Only then does it apply a controlled mutation strategy, roleplay and persona framing, base64 or ROT13 or hex or leetspeak encoding of the disallowed request, language switching, payload splitting across a prompt, markdown or comment wrapping, instruction override prefixes, unicode homoglyph and zero width smuggling, and nested injection inside data it asks the model to summarise. A refusal is never the end of a vector; it is the signal to mutate and retest against the baseline.

Phase 3: confirm, score, push, and never over claim

For every vector the agent reproduces the result with an exact curl request, captures the raw response, and then challenges its own claim before it writes it down. The status qualification rule it carries is identical to the rest of the roster and it is adversarial by design.

The rule bites in the LLM setting specifically. A payload that is merely stored or echoed back in a JSON field is not insecure output handling, because XSS needs execution in a rendered sink. A bare HTTP 200 on a reachable route is not a finding on its own. A hedged, partial answer to a jailbreak prompt is not a bypass. The agent reports every finding it identifies, but it assigns the status by demonstrated impact, so the count is honest rather than inflated.

Bounded throughout, so it never becomes the outage

An offensive LLM agent that hammered a model endpoint would be a denial of service tool, and it would also freeze its own campaign. The agent is bounded from both directions. Every command it runs carries its own end: a timeout on the shell, a --max-time on every curl, and the executor refuses anything with no natural end before it runs at all. It caps concurrency at three workers, sleeps between batches, and stops immediately on a 429, a 503 or a connection reset rather than retrying into a ban. To prove there is no rate limiting it sends exactly eleven requests and documents that all returned 200; it never floods, and it never grinds a keyspace. The design lesson the platform learned the hard way is written into the agent: one unbounded command does not just fail, it freezes everything after it and loses every finding that would have followed, so abandoning a dead end after two bounded attempts is the correct outcome.

FAQ

How does Darkmoon know an endpoint is an LLM? The orchestrator probes every responsive HTTP service for the standard inference routes and headers: /v1/models, /api/tags, /v1/chat/completions, /api/generate, the openai-* and x-ratelimit-* header families, and vLLM, TGI, llama.cpp, LM Studio and Ollama banners. A positive fingerprint dispatches the llm agent automatically.

Do I need garak for this to work? No. The agent is the attacker and the judge with curl, python3 and jq, so it has strong coverage on its own and degrades cleanly when garak is absent. garak is added for systematic, machine readable coverage, pinned to 0.16.0, and its install is non blocking so a failed install never breaks the build.

Which OWASP LLM categories does it cover? Prompt injection, direct and indirect (LLM01), insecure output handling (LLM05), sensitive information and training data disclosure (LLM02, LLM06), system prompt leakage (LLM07), excessive agency (LLM08), jailbreak and guardrail bypass (LLM09) and unbounded resource consumption (LLM04, LLM10), plus serving layer issues like unauthenticated and unmetered access.

Will it take my model endpoint down? No. Every request is bounded, concurrency is capped at three, resource exhaustion is proven with a handful of requests rather than a flood, and the agent stops immediately on a rate limit or a reset. See the OWASP LLM Top 10 case study for the eight findings this produced on a real run, and the privacy gateway write up for how your infrastructure stays hidden from the model that does the attacking.

Darkmoon is our open source project (GPL-3.0): github.com/ASCIT31/Dark-Moon, docs.