Autonomous penetration testing in your CI/CD pipeline

Most teams still buy one penetration test a year, run it against production or a staging clone, and read the PDF six weeks later when half the findings have already shipped again. Meanwhile the pipeline merges code every hour. The gap between how often you change the system and how often anyone attacks it is the whole problem, and it is the gap CI/CD security testing is meant to close. The question is not whether to shift security left, it is what you actually put in the pipeline that is worth failing a build over.

This is a methodology for wiring an autonomous agent into that pipeline. Darkmoon is a self hosted platform that plans and dispatches 50 specialist agents which run real offensive operations through a controlled MCP tool layer, and it ships with a headless CLI and a reusable GitHub Actions workflow you can call from any job. It is not a marketplace Action you click to install, it is an install script and a run command that you drop into a stage you own. The pentest agent is the entry point, and the agents it coordinates carry a status qualification model that lets you gate honestly on real findings rather than on noise.

Annual pentests versus every merge

A once a year engagement gives you depth and a human who chases the odd chain no scanner would. It does not give you coverage in time. NIST SP 800-53 lists penetration testing as control CA-8 precisely as a periodic assurance activity, and the OWASP DevSecOps Guideline argues the opposite axis, that security testing belongs in the delivery pipeline so a regression is caught at the merge that introduced it. These are complementary, not rival, positions. You keep the deep human engagement, and you add a continuous, automated check that runs on every meaningful change.

The reason this is now practical is that an autonomous agent can do more than a DAST scanner. A scanner matches signatures and reports a status code. Darkmoon actually attempts the exploit and records what it extracted, which is the difference between a build gate that fires on maybes and one that fires on proof. That distinction is the backbone of everything below.

Where the stage belongs: against an ephemeral env, not production

The single most important design choice is the target. You do not point an offensive agent at production, and you do not point it at a shared staging box that other teams depend on. You stand up an ephemeral environment as part of the pipeline, run the agent against it, and tear it down. This is the same disposable environment pattern you already use for integration tests, extended to security. The environment is built from the exact commit under test, so a finding maps to a diff, and nothing that happens during the run can touch a system anyone relies on.

Practically this means the security stage runs after your build and integration stages, once the artifact is deployed to a throwaway namespace, container network, or short lived cloud account. The agent treats that instance as the target of record. Because the environment is rebuilt every run, an exploited finding is reproducible by definition: the same commit plus the same pipeline yields the same target. This is also the honest boundary for an offensive tool. Only ever run it against infrastructure you own and are authorized to test, which in a pipeline means an environment your own job just created for the purpose.

Invoking the self hosted agent from a pipeline step

Darkmoon ships two workflow files: a caller that exposes a workflow_dispatch input for the target, and a reusable workflow_call template that does the work. The template runs on a self hosted runner, installs Darkmoon from the license gated install script, runs the pentest agent against the target string, and collects the reports the agents write inside the container. The whole stage is four steps.

# .github/workflows/template.yml (the reusable job, abridged)
jobs:
  darkmoon-headless:
    runs-on: darkmoon-runner-demo
    steps:
      - name: Download & install Darkmoon
        run: |
          curl -fsSL https://client.dark-moon.org/install.sh -o install.sh
          chmod +x install.sh
          ./install.sh "${{ secrets.DARKMOON_LICENSE_KEY }}" \
            --provider "${{ secrets.OPENROUTER_PROVIDER }}" \
            --model   "${{ secrets.OPENCODE_MODEL }}" \
            --api-key "${{ secrets.OPENROUTER_API_KEY }}"

      - name: Run pentest
        run: ./darkmoon.sh run --agent pentest "${{ inputs.prompt }}"

The prompt is just the scope, for example TARGET: 172.18.0.3, resolved from the address of the ephemeral environment the pipeline stood up. Nothing here is GitHub specific. GitLab CI runs the same install and run commands in a script: block, and Jenkins runs them in a pipeline stage. If you are wiring Jenkins in particular, keep the platform itself in scope for review too, since the same Script Console that runs your pipeline is a documented remote code execution surface, covered in our writeup on the Jenkins Script Console. The same logic applies to your forge: our GitLab security audit via the API shows what a credentialed agent finds in a self hosted instance.

Scoping and bounding the run so it never hangs the build

The two failure modes of a security stage are that it runs forever and that it fails randomly. Both are scope problems. Darkmoon addresses them at the tool layer: agents run bounded commands through an MCP backed toolbox whose executor refuses unbounded commands, so a single step cannot wander off into an open ended scan that never returns. You reinforce that from the pipeline side with a job timeout and a tightly written target string. Give the agent one host or one service, not a subnet, and let the ephemeral environment define the blast radius.

Flakiness comes from testing something that itself changes between runs, or from letting the agent reach a resource outside the disposable environment. Because the target is rebuilt from the commit and because the run is scoped to that instance, the same input produces the same findings. Treat the stage like any other test: deterministic inputs, a hard timeout, and a target you fully control. An agent that only ever touches the environment your job created cannot flake on someone else's deployment.

Gating on findings by status and severity

This is where an autonomous agent earns its place over a scanner. Every Darkmoon agent carries a status qualification model, and it is adversarial: the agent challenges its own claims before recording them. A finding is only marked EXPLOITED when the impact was executed end to end, meaning data was extracted, access was gained, or an action was carried out. It is marked CONFIRMED when the impact is demonstrated with the exact request or payload plus the raw response and the extracted data or execution trace. Anything that is a real lead but was not demonstrated is UNCONFIRMED, capped at low severity and CVSS 3.9 or below. Bare 200 responses, echoed payloads, and by design public secrets get demoted rather than trumpeted.

That gives you a gate you can defend to a platform team. Fail the build on any EXPLOITED or CONFIRMED finding at or above your chosen severity, and let UNCONFIRMED leads through as a warning that a human triages later. Because the status is derived from proof, not from a heuristic, the false positive rate of the gate is a property of the report rather than a knob you tune. A stage that only stops the line when the agent actually broke something is a stage engineers trust instead of routing around.

The report as a build artifact, with proof attached

The agents write their findings as reports inside the container, and the final step of the workflow pulls those files out and uploads them as a build artifact. The evidence, the command transcripts and the raw responses that justify each EXPLOITED or CONFIRMED status, travels with the build rather than living in someone's inbox.

- name: Collect reports from container
  if: always()
  run: |
    mkdir -p reports
    for f in $(docker exec opencode bash -c \
      "ls /opt/darkmoon/mcp/server/data/reports/*.md"); do
      docker exec opencode cat "$f" > "reports/$(basename $f)"
    done

- name: Upload reports
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: darkmoon-reports-${{ github.run_id }}
    path: reports/

The if: always() on both steps matters: you want the report even when the gate fails the build, because the report is how the developer understands why. Attaching signed, reproducible evidence to each build also lines up with what supply chain frameworks like SLSA ask for, namely verifiable provenance about what happened to an artifact on its way to release. A per commit pentest report is one more attestation about that artifact, and it is one a scanner cannot produce because a scanner has no proof to attach.

Keeping targets and secrets off any cloud model

A CI stage that reasons about your infrastructure with a language model raises an obvious concern: does your internal topology, and do the credentials the agent recovers, end up in a hosted model's context. Darkmoon's answer is the Privacy Gateway, a reversible local tokenization layer. Real values are replaced with deterministic placeholders such as IP_PRIVATE_001, HOST_INTERNAL_001 and EMAIL_001 before anything reaches the model, and they are rehydrated locally only at the moment a tool actually runs, then masked back out of every result. The model only ever sees the placeholders, never your real IPs, hosts or credentials.

We are careful not to over claim this. Deterministic placeholders still leak structure, cardinality and co occurrence by design, and the command gateway is a policy layer, not a sandbox. The full nuance is in our post on running an AI pentest without sending data to the LLM. The practical consequence for a pipeline is that you can run the reasoning against a hosted model with placeholders only, or point the same stage at a fully local model and keep the entire loop on your own infrastructure, which is the setup we describe in the local, self hosted AI pentester. For a cloud target that means the agent can work an account end to end, as in our walkthrough of autonomous cloud penetration testing, without the account's identifiers becoming a model's training adjacent context.

Wiring the stage into your pipeline

• Run the security stage after build and integration, against an ephemeral environment your pipeline creates from the commit under test and tears down when the job ends.
• Install and invoke the agent from a job step: the license gated install script, then darkmoon.sh run --agent pentest with a tightly scoped target string like TARGET: 172.18.0.3.
• Bound the run with a job timeout and a single host or service in scope. The MCP executor already refuses unbounded commands, so the stage returns instead of hanging the build.
• Parse the report and fail the build only on EXPLOITED or CONFIRMED findings at or above your severity threshold. Let UNCONFIRMED leads through as warnings for human triage.
• Upload the report as a build artifact with if: always(), so the proof travels with the build whether the gate passes or fails.
• Keep the target and any recovered secrets off a hosted model with the Privacy Gateway, or run the reasoning against a local model to keep the whole loop on your own infrastructure.

FAQ

Can you run a real pentest inside CI/CD? Yes, with the caveat that it targets an ephemeral environment, not production. Darkmoon installs headless on a self hosted runner and runs the pentest agent from a normal job step, dispatching specialist agents that attempt real exploits and write a report. It is genuine offensive testing, bounded to a disposable target your pipeline created.

Will an autonomous run make builds slow or flaky? Only if you scope it badly. Give the agent one host or service, set a job timeout, and target an environment rebuilt from the commit under test. The MCP backed executor refuses unbounded commands, and deterministic inputs against a reproducible environment mean the same commit yields the same findings, which is the opposite of flaky.

What environment should it target? A throwaway instance the pipeline stands up from the exact commit, in an isolated namespace, container network or short lived cloud account, then destroys. Never production and never a shared staging box. That is also what keeps the run authorized: you own and created the target.

How do I fail the build only on real findings? Gate on the status field. Fail on EXPLOITED (impact executed end to end) and CONFIRMED (impact proven with request, raw response and extracted data), and treat UNCONFIRMED leads as warnings. Because the agent adversarially challenges its own claims before recording them, the gate fires on proof rather than on maybes.

Does the pipeline send my code to an external model? The stage tests a running target, not your source. When it reasons with a hosted model, the Privacy Gateway replaces real IPs, hosts and credentials with deterministic placeholders before anything leaves, rehydrating locally only when a tool runs. You can also point the stage at a local model and keep the entire loop on your own infrastructure. Structure and cardinality still leak by design, so treat it as reduced exposure, not zero.

What continuous autonomous testing actually buys you

The value is not that an agent replaces your annual engagement, it is that the gap between change and attack collapses from a year to a merge. Every commit gets an offensive run against a clean copy of itself, the build stops only when the agent actually broke something, and the proof is attached to the artifact for whoever has to fix it. That is CI/CD security testing that behaves like the rest of your pipeline: deterministic, bounded, and honest about what it did and did not demonstrate. The status model is what makes it trustworthy enough to fail a build over, and the Privacy Gateway is what makes it safe to run on infrastructure you would rather no model ever saw.

Darkmoon is GPL-3.0 and self hosted: source, docs.