AI Security Testing Found Four SAML Bypasses. The Fixing Is Where It Stalls

What the model actually found, and what it did not#

On 19 August 2026, Eric Chiang, CTO and co-founder of Oblique Security, published a month of spare-time research in which he pointed an Anthropic Opus model at SAML, the ageing single sign-on protocol that still stands behind a startling share of enterprise logins. The result reads like a report from the far side of the hedgerow: four full authentication bypasses in four separate projects, two of them carrying CVE numbers. For anyone selling ai security testing, that is the headline, and the headline is the least useful part of it.

The method matters more than the mood, and in ai security testing it is the only thing worth grading. Chiang applied to Anthropic’s Cyber Verification Program to remove what he calls the guardrails on his account, was accepted, and gained access to an Opus model that could write exploits. He then ran a harness he had built himself, on a Claude Max 20x plan, across roughly a month of evenings: a gadget phase that hunts odd behaviour in the underlying XML libraries, then a findings phase that combines gadgets and confirms them with an end-to-end exploit.

Nothing here was autonomous. A human wrote the threat model, prompted the prowl, picked the quarry and did the disclosure by hand. That distinction is the first thing a buyer of ai security testing should insist on.

What survives independent checking is uneven, and an ai security testing buyer should know which half is which. Two of the four carry vendor advisories: the goauthentik advisory rates CVE-2026-57580 High at 8.7 and names the patched releases 2026.5.5 and 2026.2.6, while the litesaml advisory rates CVE-2026-63182 High at 8.1 with the fix in 5.0.1. The other two, OneUptime and the Java saml-client, rest on open GitHub issues with no CVE, no CVSS score and no published fixed version. Same sentence in the summary, very different standing in the record.

Then the correction most coverage skipped. The Authentik bug was not found by a model working alone. GitHub’s advisory credits exactly eight reporters, Chiang among them, so it is seven others plus him, and one of the eight is a team account rather than a person. The advisory went public on 15 July 2026, over a month before the research post appeared. Chiang says so himself: eight different independent researchers reported it at the same time. The simultaneity is his account rather than the advisory’s, because GitHub publishes no per-reporter dates. One of eight simultaneous finders is a fine sentence. The model found it is not.

Scope deserves the same ai security testing discipline. The Authentik advisory limits the bypass to inbound SAML Sources configured with the non-default USERNAME_LINK or EMAIL_LINK matching mode and signed assertions. Sources on the default unique-identifier mode are untouched, the SAML Provider role is untouched, and exploitation requires an attacker who already holds an account on the source identity provider and can set their own NameID.

Once achieved, the takeover is full and it persists. Read whole, that is an insider attack against a non-default configuration, not an internet-wide licence to log in as anyone. Note too that two official ratings circulate for the same bug: the advisory’s 8.7 High under CVSS 3.1, and a 9.4 Critical under CVSS 4.0 from the same numbering authority.

The denominator that does not exist#

Here is the hole underneath every hit rate a marketer might be tempted to build. Chiang writes that during his spare time he attempted to hack every SAML implementation he could find, and nowhere in the post or the research repository does he say how many that was. No denominator, no rate. Four out of what is unanswerable, so any chart, deck or advertisement implying a strike rate for ai security testing is inventing the bottom half of its own fraction. The fraction has no floor.

The twelve-project figure needs the same careful ai security testing handling. Chiang says he found signature bypasses in twelve projects on the less-watched SAML messages, AuthnRequest, AttributeQuery and LogoutRequest, and precisely one of the twelve is public: samlify issue 634, opened on 3 August 2026 and still open. His own repository README says the outstanding reports will be added as they become public. So a reader can verify one of twelve today. That is a researcher’s running count published on a company blog, not a documented industry total, and the difference matters enormously if you plan to quote it.

None of this makes the work weak. Two vendor advisories corroborate it independently, and the lightsaml write-up carries a runnable proof that ends with the bypass confirmed. It does mean reading the frame. Oblique Security sells identity security, the post closes on a demo booking, and its conclusion, that SAML is a bad protocol and teams should move to OIDC, points squarely at what the company sells. Cyber Security News picked the story up on 20 August and reproduced the four-project list faithfully, adding no maintainer comment and no denominator of its own. Syndication is not corroboration. For a security brand, that gap is the whole opportunity, and it is the gap folkfox’s cybersecurity marketing work lives in.

Finding bugs got cheap, and ai security testing has the receipts#

Set the SAML story beside the vendor telemetry and the direction is not in doubt. Anthropic ran 45 agents, one virtual machine each, a shared forum and an identical prompt, pointed at 15 open-source projects, with a separate arbiter agent deciding whether a submitted ai vulnerability was new and valid. The coordinating swarm returned 266 vulnerabilities over a 27 million token run. The standard parallel method returned 21 over 6.5 million tokens.

Read the ai security testing caveats before reaching for that ratio. This is a single vendor’s telemetry about its own unreleased models, adjudicated by another of its own agents rather than by external triage or maintainer confirmation, and Anthropic itself reports only 12 findings in common between the two methods, with roughly half the swarm’s haul sitting outside the directories the comparison arm even searched. Two different searches across two different surfaces is not a like-for-like race, whatever the ratio looks like on a slide.

The wider Glasswing ai security testing figures are bigger and no less contested. Anthropic reports its model found what it estimates are 6,202 high or critical vulnerabilities out of 23,019 in total, across more than a thousand open-source projects, and that of 1,752 high or critical findings assessed by one of six independent security firms, 90.6% proved to be valid true positives. Bruce Schneier disputes the surrounding claims on the record, calling the initiative a public-relations move, saying the superiority claim is simply not true, that something is fishy about the data, and that Anthropic will not release details. Both things hold at once: it is the best true-positive figure publicly available, and it is still the vendor marking its own homework.

Mozilla is the closest thing here to a defender-side control. It fixed 22 security-sensitive bugs in Firefox 148 using Opus 4.6, then 271 vulnerabilities in Firefox 150 after applying an early Mythos Preview. Two qualifiers travel with that second number, and both matter for any ai code security claim: the model came from Anthropic as part of a collaboration, so Anthropic cites Mozilla and Mozilla cites Anthropic, and Mozilla says it has seen no bug an elite human researcher could not have found. Scale changed. Kind did not.

Google’s ai security testing numbers need the sharpest reading of the lot. Chrome fixed 1072 security bugs across milestones 149 and 150, more than the previous 23 milestones combined, but that count mixes bugs found by AI with AI-generated fixes for bugs found by other means, and Google does not separate the two. It measures remediation throughput, not discovery yield. Anyone writing that AI found 1072 Chrome bugs has fumbled the sentence. The same post carries the other half of the squeeze: by March 2026 Chrome was receiving more bug reports than in the whole of 2025.

The academic trail runs the same way#

Outside the vendors, the ai security testing numbers hold up. TitanCA, from Singapore Management University and GovTech Singapore, reports 203 confirmed zero-day vulnerabilities and 118 CVEs from an orchestrated agent pipeline. Revelio, from a Berkeley-led group, found 19 previously unknown memory-safety vulnerabilities at around one hour per project and 300 dollars in total, and it reports a bug only when a deterministic sanitiser confirms an executable proof. Niels Provos puts a single investigation at 30 to 150 dollars per codebase at commercial pricing, and argues the discipline is an orchestration problem rather than a frontier-model problem.

Cheap enough to be routine, which is exactly why ai security testing stops being a differentiator the moment every rival can run it too. Our earlier look at benchmarked AI penetration testing made the same point from the scoring side.

One honest brake belongs here. A 2025 benchmark, Are We There Yet?, put 20 agents through 80 real-world CVEs and found the strongest performers degrade by over 33% when authentication information is incomplete, failing consistently on service-based vulnerabilities that need several components standing up together. It measures reproducing a known bug from a report rather than discovering a new one, and it predates this model generation, so it is no rebuttal of the SAML work. It is a reminder that ai security testing is brilliant in a clean repository and brittle in a messy estate, and most client estates are messy.

Fixing them did not, and the ai vulnerability backlog proves it#

Now the number the whole trade should be quoting. Anthropic says 75 of the 530 high or critical bugs it has reported have now been patched, and 65 of those have public advisories. That is the organisation whose model is credited with thousands of findings, reporting that fewer than one in seven of its serious reports has been fixed. Discovery scaled. Remediation did not, and no further ai security testing spend shifts that second number by a single bug.

Anthropic’s own write-up is refreshingly plain about why. Several maintainers asked it to slow the rate of disclosure. A high or critical bug takes about two weeks to patch on average, which is fine for one bug and impossible for a thousand. Maintainer capacity is a fixed quantity in a hedgerow that keeps growing, and every finding lands as work for a person who was already busy.

The academic framing of ai security testing has caught up with the marketing. A 2026 paper with the splendid title Demystifying the Mythos or Disrupting Bugonomics? argues that headlines emphasise capability and almost never interrogate cost and incentive, and that the near-term shift is not simply more zero-days but a squeeze on defender remediation throughput: absorbing, validating, triaging, patching and shipping. The pinch is worst in open source, where maintainer capacity does not scale with the number of agents pointed at the code.

The friction shows up inside the SAML work itself. Chiang opened the samlify issue in public because a private report went unanswered, which is a disclosure failure rather than a discovery failure. Set that beside the one fix in this cluster whose mechanism is documented: goxmldsig patched a quadratic memory amplification in v1.6.1 by capping traversal at 1000 nodes, after an unauthenticated request of roughly 60 KB carrying a non-exclusive canonicalisation algorithm could drive a process to gigabytes of resident heap. Somebody had to read that, reproduce it, design the cap and ship it. No model did that part.

Ask a working vulnerability researcher where the hours actually go and the answer is never the clever bit.

Harness setup, logging, symbol resolution, cross-referencing CVE records against the exact build in front of you: that is the half of the job discovery tooling does not touch, and it sits on the same desk that has to triage whatever the agents throw over the fence. Sell into that desk and you are selling to somebody whose scarcest resource is attention, not tooling. Sell more findings into it and you are the problem.

Anyone tracking the distance between disclosure and patch has walked this trail before: we wrote about the gap between Patch Tuesday and exploited Wednesday earlier this year. What is new in 2026 is that the supply side of the problem became cheap while the demand on defenders stayed exactly as expensive as it always was.

The slop panic that quietly resolved itself#

For two years the standard rebuttal to ai security testing was a single word: slop. It had evidence behind it. In July 2025 Daniel Stenberg reported that about 20% of curl’s inbound security submissions were AI slop and about 5% were genuine, at roughly two reports a week, and he was visibly weighing what to do about a bounty programme that was costing more attention than it bought.

In January 2026 he acted. The curl bug bounty closed on 31 January, after 87 confirmed vulnerabilities and over 100,000 USD paid out since 2019. The confirmed rate had fallen from north of 15% to below 5%. Read that post properly and he names three causes, only one of which is AI: slop, humans submitting worse reports, and reporters who want to poke holes rather than help. HackerOne cohort data in the same post shows curl’s inbound volume climbing sharply across four quarters while Ruby, Node and Rails stayed flat or drifted down.

Then, in April 2026, the story turned, and this is the part most coverage has not caught up with. Stenberg reported that curl returned to HackerOne in March 2026, that report frequency runs at roughly double the 2025 rate, and that the confirmed-vulnerability rate is back to and even surpassing the pre-AI level, somewhere in the 15-16% range. Almost every security report now uses AI to some degree, he writes, and they are mostly very high quality. He projects curl may publish closer to 50 vulnerabilities in 2026.

Volume is the durable finding, not slop#

So the lasting ai security testing shift is not slop. It is sheer volume. A maintainer’s inbox doubled and the quality held, which is a better problem and still a problem, because the work at the far end of the pipe is unchanged. Note too that his supporting list of other projects seeing the same pattern came from an unscientific poll on Mastodon, his own word for it, so treat that list as anecdote rather than survey.

For any vendor writing about ai security testing in 2026, quoting the 2025 slop posts alone is now a factual error with a date stamped on it. The primary source moved. Anyone still selling defence against a wave of worthless AI reports is guarding last year’s hedgerow while the trail has plainly turned, and a buyer who reads the April post will notice within a minute.

What to say now that ai security testing is table stakes#

Every vendor in this market is about to say the same sentence: we use AI to find more vulnerabilities. It will be true, and it will be worthless, because the thing it describes just got cheap enough that a CTO did it in his evenings for the price of a subscription. Positioning built on a capability everybody suddenly has is not positioning at all, it is noise wearing a lanyard.

The scarce good sits on the other side of the ai security testing report. Triage that ranks thousands of findings by what an attacker can actually reach. Remediation that ships a patch rather than a ticket. The judgement to separate a real finding from a plausible one, which is precisely why the Revelio team constrained its pipeline to bugs carrying a reproducible proof, and why Provos insists a proof of concept is required for defenders because findings from static analysis alone produce a high volume of false positives. Sell the part that stayed hard.

Then prove your ai security testing in public. The only reason this article can be written at all is that four artefacts are readable by anyone: two GitHub advisories, one open issue and a research repository. A security brand that publishes its own triage record, its own patch rate and its own denominator will outrank a competitor claiming a bigger number with nothing underneath, both with buyers and with the answer engines now doing the summarising for them. That is the same discipline behind our SEO and GEO work and our brand strategy practice: say the checkable thing first, and say it before the trade press does.

There is a compliance clock ticking behind all of this. We covered AI security growing a deadline before it grew evidence, and this morning’s NIST guidance on AI prompts for framework work hands security teams written language for exactly this sort of task. Both are reasons a buyer will ask a harder question this quarter than they asked last, and a brand that has rehearsed the honest answer will enjoy the conversation more than one that has not.

The fox rule applies. Do not sell the sprint, sell the sieve. Finding is a cheap thicket to walk now; sorting the shards that stay in the mesh is the scarce skill, and it is the only story in this category worth telling in 2026. If you want that story built rather than merely described, our content marketing services start where this article started: with the primary document, read properly, before anyone else has quoted it.

Frequently asked questions#