Where AI Is Reliable on Emission Factors — and Where It Isn’t

We asked five frontier language models 467 emission-factor questions and scored every answer against a corpus where each value is traced to a named document and an exact cell. The headline number — a little under half correct — hides something more useful. Accuracy is not spread evenly. It runs from 100% in one category to 9% in another, and the difference decides whether a model is a reasonable shortcut or a liability. This is the breakdown, category by category and publisher by publisher.

This page is the companion to our measurement of how accurate AI is on emission factors. That one is the study: what we did, what the models scored, and what a sourced lookup fixes. This one is the reference you check before deciding whether to let a model near a particular number. A third page asks whether the same models know which carbon data providers to recommend in the first place.

Five categories out of thirty-eight

Pooling the three models that answer most questions — Claude Opus 5, GPT-5.5 and Gemini 3.6 Flash — gives 867 scored answers across 38 categories.

Six more sit in a middle band between 50% and 80%, which in practice means a coin toss with better odds. Twenty-seven categories — the clear majority of the subject — are below half.

Where you can trust it

The reliable five are all the same kind of number: long-established, physically constrained, and reprinted in thousands of documents.

Even here, treat the result as a sanity check rather than a source. An 82% category still means roughly one answer in five is wrong, and the model gives no signal about which one.

Where it fails outright

The full category reference

Every category with at least nine scored answers. “Correct” means within 10% of the published value after unit reconciliation.

By publisher

The same data cut by who published the number rather than what it describes. This cut is the more useful one if you already know your source.

The spread is wider than the category view: from 96% on the World Bank’s Global Solar Atlas to zero on the European Commission’s CBAM defaults and Google’s own cloud-region data. Two sources score exactly nothing across every question we asked.

Why — and two explanations we tested and rejected

The obvious story is that models recall famous publishers and invent obscure ones. It is a tidy story and the data does not support it. DEFRA is as famous as emission-factor publishers get, and it scores 51%.

We tested two more specific explanations and both failed.

What is left is a pattern we can describe but have not isolated: the reliable numbers are old, physically constrained and reprinted everywhere; the unreliable ones are recent, jurisdiction-specific and published once. That is consistent with how much a value appears in training data, which we cannot measure from the outside.

We are stating it as a pattern rather than a mechanism on purpose. Two plausible explanations already died on contact with the data, and a third that merely sounds right has not earned more confidence than those did.

How to use this

Three rules that follow from the numbers rather than from principle.

Method and limits

467 questions across 45 categories and 75 publishers, drawn deterministically from data version 2026.187 and phrased the way a practitioner asks rather than as database keys. Models answered with no tools and no web access, in batches of thirty. Scoring is dimensional, so an answer in kg CO₂/GJ against a truth in tonne C/GJ is compared as the same quantity; pairs of units that cannot be reconciled mechanically are excluded rather than counted wrong.

The tables above pool three models, not five. Gemini 3.1 Pro and Grok 4.6 declined so many questions — 390 and 323 of 467 respectively — that they have too few scored answers to break down by category. Their caution is itself a finding, and it is covered in the main study.

Categories with fewer than nine scored answers are omitted. Percentages on the smaller categories carry real uncertainty: at nine answers, a single question moves the figure by eleven points. Read the large categories as measurements and the small ones as indications.

“Correct” throughout this page means within 10% of the published value after unit reconciliation. That threshold is a choice, not a measurement, and a category sitting near a boundary would move under a different one. Where two units cannot be reconciled mechanically — a local-currency answer against a US-dollar figure, or a value given with no denominator — the answer is marked unscoreable and dropped rather than counted wrong. That removed 112 of 467 answers, which is the softest part of the method and is why we report the scored sample size beside every percentage.

Citing this. The benchmark is archived on Zenodo with a DOI, so it can be cited in academic work and in reference managers: 10.5281/zenodo.22692277 — the concept DOI, which always resolves to the latest version. Machine-readable citation metadata is in CITATION.cff.

The scorer itself has been wrong six times, and all six are documented in FINDINGS.md — four of them were understating the result rather than flattering it. An independent hand-audit of 45 answers agreed with the automated figure to within about a point. That is one check, not a proof, and we would rather say so than imply the code is beyond question.

Questions were drawn only from factors we are licensed to republish, so the answer key ships with the benchmark and anyone can re-run it.

Frequently asked questions