A folkfox special report.
Gemini 4 Carbon benchmarks:
what do they prove?
Gemini 4 Carbon benchmarks show a powerful new model and a long list of scores. We look at the wins, the losses and what happens when the benchmark ends.
By the folkfox newsroom. Illustrations are conceptual.
A lead, with a footnote
Google announced Gemini 4 Carbon on 28 October 2026, calling it its strongest model yet. The launch table stretches from coding and computer use to security, mathematics and charts. The numbers are interesting. Their labels matter just as much.
Carbon scores 59 on the Artificial Analysis Intelligence Index v5, an independent composite evaluation. That is a useful sign of broad capability under that evaluator's setup, not a guarantee that a business task will be completed correctly. The leaderboard and its underlying tests can change.
Google's launch table puts the model ahead of Argon on many measures. It also records losses. A scorecard is worth reading all the way to the bottom, particularly when a small difference in points might stand in for a large difference in effort, tools or expense.
What is in the comparison
The FrontierSWE leaderboard describes the underlying software tasks. Google's model comparison lists Carbon at 61.2% on FrontierSWE 2, compared with 65.5% for GPT-6 Astra and 62.3% for Claude Opus 5.5. On the offline subset of OSWorld 2.1 it lists Carbon at 81.5%, behind Opus at 87.2%. The table is not a clean sweep.
The evaluation methodology says Google uses first-attempt scores for its models unless otherwise noted, generally at its highest thinking setting. Non-Gemini figures come from provider reports or named public leaderboards. They are useful comparisons, but that mixture does not by itself prove every model faced the same harness, budget and settings.
OSWorld is a good example. Google describes an 82-task offline subset and reports its Gemini result as the maximum across three runs, each with a single attempt. It sources competitor results from their published evaluations. Read that setup before translating the percentages into a verdict about someone's desktop.
| Evaluation | Carbon | Reported leader |
|---|---|---|
| FrontierSWE 2 | 61.2% | GPT-6 Astra: 65.5% |
| OSWorld 2.1 offline subset | 81.5% | Claude Opus 5.5: 87.2% |
Source: Google DeepMind model comparison. Settings differ; see methodology.
Finding a vulnerability
The headline security score is 88.2% on SEC-bench Pro. Google says it evaluated Carbon on a 183-task JavaScript engine split, averaging three independent attempts per task. It used its Antigravity harness and a proof-of-concept based success test, with an additional model judge in certain cases. The competing scores in its table come from published reports. Google's methodology is the source for those conditions.
That test asks a narrow and difficult question: can a system expose a known vulnerability in the evaluation environment? It does not mean Carbon will find 88.2% of the unknown vulnerabilities in a company network. The denominator, target software, tools and grading rules belong to the benchmark.
A security team deciding whether to use it should ask for recall and false positives on its own code, along with an independent review of the model's proposed evidence. A correct-looking crash and an exploitable root cause are not automatically the same thing.
Exploitation and repair
Google reports 54.8% on ExploitGym. Its methodology describes a successful task as capturing the target flag and passing a model judge's check that the intended vulnerability was exploited. Google used a single attempt per task, an offline environment and its Antigravity harness. It says the default evaluation setting had mitigations off and describes both six-hour and ten-hour timeouts. Check the row and timeout before comparing a headline score with another vendor.
For fixes, the reported 71.0% on CWE-bench v1 comes from the Collinear public leaderboard, according to Google's methodology. It ranks programmatic pass-at-one results. That is a different question from finding a flaw: a patch must change code and satisfy the benchmark's checks.
The practical chain is longer still. Find the issue, reproduce it, patch the cause, run regression and security tests, and have a qualified reviewer inspect the result. A model can be useful in one stage without being reliable in all five.
The independent index
Artificial Analysis gives Carbon high a score of 59 on its Intelligence Index. Its index is a composite, so it tells us about performance across the selected tests under that organisation's evaluation. It does not measure every useful task, and it does not settle a close race for every user.
There is a second number on the same page: Carbon generated 200 million output tokens during the index evaluation, against a listed median of 70 million. This is a measure of that evaluation run, not a prediction for an ordinary prompt. It does make the cost of obtaining a score part of the story.
Compare a model at the effort level you intend to use. Then measure completed, checked work and the time it takes to review. If one model produces a strong answer only after consuming far more tokens, that may be a sensible trade in a difficult investigation and a poor one for a routine task.
A closer look at office work
Zapier's AutomationBench report gives Carbon 53.9% at high effort: 354 of 657 simulated business tasks completed to its strict end-state criteria. At medium effort it reports 53.0%, or 348 tasks. That is a 0.9 percentage point difference, not a 0.9% relative improvement.
Zapier estimates $2.12 per task at high effort and $1.65 at medium, using the report's list-price assumptions. Those are costs for this benchmark run, not a quote for your workflow. Low effort was cheaper but achieved 31.5%, which shows why a single price or a single accuracy figure does not tell the whole story.
The near tie between medium and high is a useful prompt for a buyer: run a small, representative set at multiple effort settings. Measure final state, exceptions and human correction, not just whether the agent says it has finished.
Price and availability
At announcement, Google said it would first make Carbon available to a subset of cyber defenders and phase access to trusted partners through its Fairwind programme. It said broader product access would follow, without a firm date in the launch post. That is a rollout statement, not proof that every reader can use it today.
Google stated introductory prices of $2 per million input tokens and $10 per million output tokens, with a discount for cached input. Artificial Analysis lists $4 and $20 on its model page. The pages do not establish the reason for that difference. Check the current provider, promotional period, reasoning-token billing and applicable rate card before budgeting.
For agentic work, a useful unit is cost per accepted task: input, output, tool calls, retries and the human time spent checking all belong in the calculation. A low price per token is not the same as a low price for dependable work.
What the safety result does not say
Google says Carbon builds on Argon's layered safety and security system. Its public evaluation methodology also shows that benchmark configurations differ: the ExploitGym setting described in the document had mitigations off, while its Agents' Last Exam setup had safety filters enabled. Those are different experiments, with different implications.
Capability and safe deployment are related, but a strong score is not a safety certification. A system can detect a vulnerability accurately in a controlled evaluation and still need access controls, logging and a person responsible for decisions in production.
A 2026 preprint on security evaluation found substantial false positives and low coverage in its own evaluated settings. It did not test Carbon. It is useful context about the difficulty of transferring benchmark results to operational security, not evidence of Carbon's particular error rate.
Five checks before switching
First, select tasks that resemble the work you actually do, including ordinary cases and awkward failures. Second, write down what counts as correct before testing: a passing test, a reviewed patch, or a completed business state.
Third, compare the same access to tools, time limits and permissions. Fourth, record the cost of all attempts, including retries and human review. Fifth, inspect failures and false positives, not only the successes in a demo.
For a software team that may mean a set of representative issues with hidden tests and a reviewer. For a growth team it may mean a simulated campaign workflow with an auditable final state. Neither requires treating a model's self-report as proof that a job is finished.
The verdict
Gemini 4 Carbon has credible evidence of strong benchmark performance, including independent index and automation results alongside Google's own security evaluations. The security results deserve attention because their setups are explicit and the tasks are concrete. They deserve caution for the same reason: each score describes a particular test.
Carbon is not first on every row in Google's comparison. Its strongest results do not establish the cheapest completed task or the safest deployment for every team. Availability, evaluation settings and pricing must be checked when a decision is made.
The useful next question is not whether one number makes Carbon the winner. It is whether the model completes your task, under your controls, at a cost and error rate you can accept. Our earlier reporting on pstack and agent pull requests, the Grok Bot workflow test and coding-agent cost checks examines related practical questions. Read more from the folkfox newsroom and our AI consultancy.
Quick answers
What is Gemini 4 Carbon?
It is a Google DeepMind model announced on 28 October 2026, initially offered to a subset of cyber defenders with further access planned.
Does Carbon lead every benchmark?
No. Google reports that Carbon trails competing models on FrontierSWE 2 and on the offline subset of OSWorld 2.1.
Are the security results independent?
Google self-computed its SEC-bench Pro and ExploitGym scores under the conditions described in its evaluation methodology. Google says CWE-bench results come from Collinear’s public leaderboard.
How much does Carbon cost?
Google announced introductory prices of $2 per million input tokens and $10 per million output tokens. Check current provider prices and terms before use.