The Score Everyone Is Quoting Belongs to a Different Benchmark.

What Google actually announced#

On 2 September 2026 Google published two posts. One introduced Gemini 3.8 Flash and 3.8 Flash Cyber. The other announced Fairwind, a programme that gates the cyber model behind vetted access, with, in Google's words, more than 650 participating partners globally. Participation is limited to governments and national cyber authorities, critical infrastructure operators in healthcare, telecoms, energy and financial services, and core technology platforms.

The programme terms require participating organisations to restrict access to internal cybersecurity, incident response and penetration testing teams, and to deploy protections including multi-factor authentication. Google also stated the commitment brings its total cybersecurity funding above $100 million globally, including $36 million for 35 cyber clinics supporting more than 1,250 hospitals, public school districts and municipal utilities in the United States.

That is a defensible way to ship a capability that cuts both ways, and it deserves credit. The part that has travelled badly is the benchmark number attached to it.

Gating matters commercially as well as ethically. When ai security tools are available only to vetted partners, the ordinary mechanism by which a market learns what a product is worth, which is customers comparing notes, simply does not run. Buyers are left with the vendor's own numbers and each other's guesses, and that is the environment in which a misread benchmark can circulate for a week unchallenged.

The 47.2 per cent belongs to a different benchmark#

The figure circulating this week is that Gemini 3.8 Flash Cyber scored 47.2 per cent on CyberGym. It did not. Read Google's own post carefully and there are two separate paragraphs. The CyberGym paragraph carries no number at all: it claims frontier-level performance in autonomous vulnerability discovery, with no percentage, no level, no trial count and no named comparison. The 47.2 per cent belongs to the next paragraph, and to CWE-Bench, an external patching benchmark run by Collinear AI.

These measure genuinely different things. CyberGym, published by researchers at UC Berkeley, asks an agent to reproduce a known vulnerability with a working proof-of-concept across 1,507 real-world vulnerabilities drawn from 188 OSS-Fuzz projects at pre-patch commits.

CWE-Bench asks something else entirely: audit and patch, across 100 held-out tasks covering 54 distinct weakness classes in six languages. Finding a hole and fixing a hole are not the same skill. Conflating them is how ai security tools get bought on the wrong evidence, and it is the single most common error in this category of purchase.

Second place, and the honest reason it still matters#

On CWE-Bench's own leaderboard, Gemini 3.8 Flash Cyber's 47.2 per cent sits behind Claude Fable 5 at 47.8 per cent. Google did not name the model above it, and framed its result as sitting on the Pareto frontier, which is a claim about cost rather than capability. On the published cost column that framing holds up: Google's entry runs at $3.64 average cost per rollout against $10.27 for the model half a point ahead of it.

Cost-per-rollout is a legitimate axis on which to compare ai security tools, and arguably the most honest one, because it is the axis a buyer actually pays on. So the accurate sentence is narrow and still useful: on a patching benchmark, Google's cyber model is a fraction behind the best result at about a third of the cost. The inaccurate sentence, the one now in circulation, is that it leads a vulnerability-discovery benchmark. It does not appear on that benchmark's public leaderboard at all.

What the CyberGym leaderboard actually shows#

The public CyberGym leaderboard, maintained as part of a frontier AI cybersecurity observatory, carries forty ranked entries. Gemini 3.8 Flash Cyber is not among them. The scores at the top are also on a completely different scale to 47.2 per cent, which is the clearest signal that the two numbers were never comparable.

This is the practical reason to care about which board a number came from. Anyone assembling a shortlist of ai security tools from vendor posts alone will end up comparing a patching score against a discovery score and calling it a ranking. None of this makes Google's model bad, and none of it makes Google dishonest. Its post is defensible read slowly. It is simply written in a way that survives a careless read badly, and careless reads are what procurement decisions are made of. For anyone building an ai vulnerability management case internally, the lesson is procedural rather than editorial: ask which benchmark, ask what the benchmark asks the model to do, and ask where the model sits on that benchmark's own public board.

CodeMender and the missing evaluation#

Most ai security tools ship with a companion agent these days, and this one is no exception. Paired with the model is CodeMender, DeepMind's agent for finding and fixing code security defects, which entered preview on Google Cloud six weeks before the 3.8 Flash Cyber launch. Its only published output figure is that it has upstreamed 72 security fixes to open source projects over six months, to codebases as large as 4.5 million lines.

That gap matters more than it sounds, because the entire commercial promise of ai code security, and of ai security tools generally, is throughput: not that a machine can write a correct patch, which is established, but that it can write enough correct patches, at a low enough false-positive rate, that a human review queue does not become the new bottleneck. The published 72 tells you nothing about that ratio. CodeMender is also human-gated rather than autonomous, which is the right design and also the thing that caps its throughput.

None of that is an argument against the tool. It is an argument about what a buyer can currently know, which is very little, and about what the market has quietly accepted as sufficient evidence. Most ai security tools now reach procurement with a benchmark number, a case study and no denominator anywhere in the pack. The organisations that get value out of them tend to be the ones that ran their own evaluation on their own codebase before signing, which is expensive, unglamorous, and the only thing that actually works.

Behind all of this sits a thicket of overlapping public catalogues that nobody markets and everybody relies on. The weakness classes CWE-Bench tests against are MITRE's. The vulnerabilities CyberGym reproduces come from real projects with real disclosure histories. A vendor claim that cannot be traced back to that undergrowth is a claim with nothing underneath it.

That traceability is the whole test. Every serious number in this article came from a page anyone can open: two Google posts, two public leaderboards, one arXiv paper and one product page. Nothing here required access to the gated model, which is fortunate, because almost nobody evaluating ai security tools this quarter will have it.

The price doubles in January#

The pricing is the least ambiguous part of the announcement and the part most likely to matter to a budget. Gemini 3.8 Flash is available at the same introductory price as 3.7 Flash, $0.75 per million input tokens and $3.75 per million output tokens. A footnote sets the introductory price to expire on 31 December 2026, with the rate rising from 1 January 2027.

Read the three facts together and a shape appears, one that recurs across ai security tools as a category. Access is gated, so most buyers cannot evaluate the tool independently. The headline benchmark is widely misattributed, so the public comparison is wrong. And the price steps up at a fixed future date, so any pilot costed at today's rate is costed against a number that expires. That combination is not unique to Google. It is becoming the standard shape of how ai security tools reach market, and it is exactly the shape a procurement process should be built to resist.

What to ask before you buy AI security tools#

A fox does not judge a burrow by the size of its entrance. It reads the ground around it, checks how many ways out there are, and notices whether the earth at the mouth has been freshly turned. Benchmark scores are the entrance. These are the ground.

The instinct worth borrowing is patience. A vulnerability claim is a scent, not a quarry, and the distance between the two is where most procurement money is lost. Follow the trail to the benchmark, then to the leaderboard, then to the denominator, and only then decide whether the thing at the end of it is worth the price that starts in January.

The checks below are cheap, and they are the difference between buying ai security tools on evidence and buying them on presentation. Ask which benchmark a number came from, and what that benchmark asks the model to do. Ask where the vendor sits on that benchmark's own public leaderboard, not on the vendor's chart. Ask for the denominator behind any count of fixes, findings or detections. Ask when the quoted price expires. And for gated tools, ask what independent evaluation exists at all, because access restrictions that are entirely justified on safety grounds have the side effect of making independent ai threat detection comparisons impossible.

None of that requires distrusting the vendor. Google's own post contains every correction in this article; it simply requires reading it slowly rather than reading the summary. Public catalogues like MITRE's CWE list, the National Vulnerability Database and CISA's known exploited vulnerabilities catalogue remain the neutral ground on which any ai cybersecurity claim can be checked, and they cost nothing.

One closing note on timing, because it is the detail most likely to bite. A pilot begun this month runs at the introductory rate and reports a cost per finding that will not survive January. If that pilot is the business case, the business case is wrong by construction. Cost the trial at the post-expiry rate, and if the numbers only work at the introductory price, that is not a pilot result, it is a discount.

Frequently asked questions#