OWASP Juice Shop
A teaching application with many vulnerability categories.
home / benchmark
public benchmark
Benchmark: AI penetration testing vs automated scanners
The benchmark compares the system with well-known scanners on deliberately vulnerable test applications. The method is written before the results.
Status: in preparation. No results are published yet. We do not cite percentages or comparisons until they are measured and published here.
targets
Vulnerable applications, known and public.
DVWA
An application with adjustable difficulty levels.
WebGoat
Guided lessons on access control and logic.
Mutillidae
Classic vulnerabilities and authorisation flaws.
comparison
Against the scanners many already use.
OWASP ZAP
Web application scanner, active and passive.
Nikto
Scanner for web server configurations and versions.
Nuclei
Template-based scanner with an open library.
static scanners vs AI
Burp Suite, Acunetix, ZAP, Nuclei: strong on the known, blind to the new.
Scanners work from databases of signatures, templates and rules: they find what someone has already described. AI-assisted testing can use the same tools and also reason about the application: understand roles, form hypotheses, chain steps. That is what a scanner does not see, and what often escapes even a human reviewer with plenty of time.
Scenario estimate, not a measurement. The percentages below are a capability model per defect class, built from the known characteristics of the tools. They are not benchmark results: those will be published here only after measurement, with commands and versions.
Estimated coverage by vulnerability class
CVEs and known signatures
Nuclei92%
Acunetix88%
Burp Scanner84%
ZAP72%
AI + tools94%
Misconfigurations
Nuclei80%
Acunetix78%
Burp Scanner70%
ZAP65%
AI + tools90%
Classic injection (SQLi, XSS)
Nuclei55%
Acunetix85%
Burp Scanner88%
ZAP70%
AI + tools92%
Access control (IDOR/BOLA)
Nuclei8%
Acunetix12%
Burp Scanner18%
ZAP10%
AI + tools78%
Business logic
Nuclei2%
Acunetix4%
Burp Scanner6%
ZAP3%
AI + tools64%
Multi-step chains (auth, race, state)
Nuclei1%
Acunetix3%
Burp Scanner5%
ZAP2%
AI + tools58%
Where scanners are enough
Signatures, known CVEs, headers and configuration: fast, repeatable, cheap. AI-assisted testing uses them, it does not replace them.
Where reasoning is needed
Access between roles, business rules, state and sequences: there is no signature to look for, the application has to be understood.
Where the human stops
Thousands of combinations of role, endpoint and state: the test tries them all, at every release, without tiring.
October 2026
October 2026: bug bounties close under the weight of automated reports.
An honest reading: programs are not stopping because too many real critical vulnerabilities arrive, but because too many unverified automated reports do. That is why proof matters: reproducible evidence, not a hypothesis.
- Early 2026 · curl ends its HackerOne bounty, swamped by model-generated false positives.
- March · Bugcrowd reports more than four times the submissions in three weeks, mostly fake.
- April · Nextcloud suspends its paid programme.
- September · Intel stops paying cash rewards, then pauses the programme.
- 1 October · Google pauses the Open Source VRP for product vulnerabilities, until at least Q1 2027.
What follows. Those receiving reports need less volume and more proof. That is why we deliver only verified defects, with steps to reproduce them, and human review stays in the process. Sources: Malwarebytes, BleepingComputer, TechTarget, Decrypt.
metrics
What we measure, with the same rules for everyone.
| Metric | Meaning |
|---|---|
| True positives | Real vulnerabilities, manually verified on the target |
| False positives | Reports that do not correspond to real vulnerabilities (false positives) |
| False negatives | Known vulnerabilities of the target not detected |
| Found only by us | Real vulnerabilities not detected by the comparison scanners |
| Time | From start to delivery of the list |
| Review hours | Human time spent verifying the result |
publication
We publish even the weak results.
Method before data
The method is published before each measurement and does not change afterwards.
Commands with results
The commands and versions used, so the measurement can be repeated.
Where we lose
If a scanner finds something we do not, we write it down.