· 8 min read
Two European frameworks have changed how a lot of organisations have to think about security testing. The NIS2 Directive widens the set of entities that must manage cyber risk and demands that they assess whether their measures actually work. DORA sets a digital operational resilience regime for the financial sector, with resilience testing at its core. Neither is satisfied by a report from twelve months ago describing a system that no longer exists. This piece is about why point-in-time testing ages badly under these regimes, and where repeatable, autonomous testing fits, without pretending it replaces the human work these rules require.
Not legal advice
This is a security engineering perspective, not legal or compliance advice. NIS2 is transposed into national law by each member state and the exact obligations depend on your sector, size and classification; DORA and its regulatory technical standards define who is in scope and what testing is mandatory. Confirm your specific duties with qualified counsel and your competent authority before relying on anything here.
What the two regimes actually ask for
Strip away the acronyms and both frameworks point in the same direction: security is a process you keep proving, not a certificate you frame once. NIS2 (Directive (EU) 2022/2555) requires in-scope essential and important entities to take appropriate measures and, critically, to have policies and procedures to assess the effectiveness of those measures. DORA (Regulation (EU) 2022/2554, applying from January 2025) requires financial entities to run a digital operational resilience testing programme, and for the most significant ones it mandates advanced threat-led penetration testing on a multi-year cycle.
The word that matters in both is effectiveness. You are not being asked to own a firewall, you are being asked to show, on an ongoing basis, that your controls hold up. That is a much higher bar than a single annual scan, and it is a bar that decays the moment your environment moves.
Why one audit a year decays
A yearly pentest produces a snapshot of one week. Everything that changes in the other fifty-one weeks is untested until the next one. In a modern estate that is most of the attack surface:
- Deploys. A team that ships daily changes its own code hundreds of times between two annual audits. Each deploy can introduce, or reopen, an exploitable path.
- Infrastructure drift. A new bucket, a widened security group, a rotated but over-scoped key, a fresh Kubernetes namespace. None of these were in scope last January because none of them existed.
- Dependencies and identities. Third-party components and the accounts that reach them change constantly. The chain that matters this quarter is rarely the one someone tested last year.
- New disclosures. A technique that was theoretical at audit time can be trivially exploitable a month later. A frozen report never revisits that.
None of this is an argument against the annual engagement. It is an argument that the annual engagement cannot be the only engagement if the standard you are held to is continuous effectiveness rather than a single date.
What continuous testing adds, and what it does not replace
An autonomous pentester is well suited to the repeatable half of this problem. It can re-run the same disciplined attack chains across your estate as often as you like, without getting bored or cutting corners on the twentieth account, and it can gate a build on findings so that a regression is caught the day it ships rather than eleven months later. That is coverage and cadence, which is exactly what a point-in-time model lacks.
| Dimension | Annual human pentest | Continuous autonomous testing |
|---|---|---|
| Cadence | Once or twice a year, scheduled | On every change, on a schedule, or on demand |
| Coverage over time | One week, then frozen | Re-run across the estate as often as needed |
| Creativity and business logic | High: a human reasons about intent and abuse of legitimate flows | Bounded: strong on known chains, weaker on genuinely novel logic |
| Regulatory role | Satisfies mandated red team / TLPT-style requirements | Evidence of ongoing effectiveness testing between engagements |
The honest framing is complementary, not either-or. DORA's threat-led testing and the deep manual red team engagements in a mature NIS2 programme are human work, and we are not suggesting an autonomous tool discharges those obligations. A skilled tester reasons about business logic, chains social and technical steps, and judges intent in ways a bounded automated agent does not. What autonomous testing does is fill the long gaps between those engagements with repeatable, evidence-producing runs, so the annual test starts from a surface that has already been kept honest.
Evidence a regulator or auditor can read
Effectiveness testing is only useful to a compliance file if it produces evidence rather than adjectives. The value of proof-of-exploitation reporting here is direct: each finding is graded by what was actually demonstrated, EXPLOITED when impact ran end to end, CONFIRMED when the exact request or payload and the raw response back it, and a capped low severity when there is a lead but no demonstrated impact. That distinction is exactly what an assessor wants to see, and it is far more defensible than a wall of unverified scanner output. We wrote about why in why proof of exploitation beats AI vulnerability scores.
Two more properties matter for regulated data. Running the loop on a local, self-hosted model keeps target detail on your own infrastructure, and the privacy gateway replaces real IPs, hostnames and credentials with deterministic placeholders before anything reaches the model, so the model works from markers rather than your production values. For an organisation whose whole reason for testing is a data-protection and resilience regime, sending that same data to a third-party model to test it is an awkward contradiction.
A practical cadence
- Per change. Wire autonomous testing into CI/CD so an exploited finding fails the build. Our CI/CD write-up shows the pattern.
- Scheduled sweeps. Re-run the full estate on a regular cycle to catch drift and new disclosures, and keep the reports as your effectiveness evidence.
- Human engagements on top. Keep the mandated red team and threat-led tests, now starting from a surface that continuous testing has already exercised.
The point
NIS2 and DORA moved the goalposts from "did you test once" to "can you show your measures keep working". A single annual audit answers the old question, not the new one. Continuous autonomous testing is how you keep answering the new one between the human engagements that the regulations still, rightly, require. Darkmoon is GPL-3.0 and self hosted: source, docs.