Training vs Inference (AI Compute) — Definition and GHG Accounting Context
The carbon story people tell about AI is a training story: the eye-watering energy of teaching a giant model, spent once, in a burst. It is a real cost — but it is also a distraction. The model then goes on to answer questions, billions of them, each sip of energy tiny but never-ending. Add those sips up over a popular model’s life and they usually swamp the training that made headlines.
AI compute has two phases, and they carry carbon in opposite ways. Training is the one-time cost of building a model; inference is the per-query cost of running it.
Training is the compute that builds an AI model — concentrated, one-time, and large (a single large language model can take on the order of a thousand megawatt-hours). Inference is the compute that runs the finished model to answer each query — tiny per query, but paid every time it is used. Because popular models are queried billions of times, cumulative inference usually overtakes the whole training cost within weeks and dominates lifetime emissions. The split mirrors embodied versus operational carbon.
Definition — The Two Phases of AI Compute
An AI model’s life has two compute-intensive phases, and telling them apart is the key to understanding its carbon. Training is the process of building the model: feeding it vast data and adjusting billions of parameters over days or weeks on large clusters of accelerators. It happens once (or occasionally, when a model is retrained), and it is enormously energy-intensive while it runs. Inference is the process of using the trained model: each time it answers a prompt, classifies an image, or returns a search result, it performs a much smaller burst of computation.
The two differ not in kind — both are electricity turned into computation in a data centre — but in shape. Training is a single tall spike of energy; inference is a long, low plateau that lasts as long as the model is in service. That difference in shape is what makes their carbon behave so differently, and why quoting one without the other gives a misleading picture of an AI system’s footprint.
This split has a familiar parallel elsewhere in carbon accounting: it is the AI version of embodied versus operational carbon. Training is like embodied carbon — front-loaded, fixed, spent up front to create the asset. Inference is like operational carbon — spread over the asset’s working life and scaling with how much it is used.
Definition at a glance
| Training | Inference | |
|---|---|---|
| What it is | Building the model | Running the model per query |
| When | Once (or on retrain) | Every use, for the model’s life |
| Per-event energy | Huge — a single large run | Tiny — a fraction of a watt-hour |
| Scales with | Model size & training data | Number of queries (usage) |
| Carbon analogue | Embodied — front-loaded | Operational — spread over use |
Training: The Concentrated One-Time Cost
Training is the number that makes headlines, and the headlines are not wrong about its size. Training a frontier model consumes the output of a large data-centre cluster running flat out for weeks. In the most-cited reference figures, from Patterson and colleagues’ 2021 study, training GPT-3 drew on the order of 1,287 MWh of electricity and produced roughly 552 tonnes of CO₂e — comparable to the annual emissions of dozens of average people, spent to build a single model.
Two things about that number matter for accounting. First, it is fixed and one-time: once the model exists, that carbon is spent and will not recur unless the model is retrained. Second, it already embeds the data-centre overhead — the PUE of the facility and the carbon intensity of its grid are baked into the reported total, which is why the same model trained on a cleaner grid would show a far smaller figure for identical computation.
Inference: The Per-Query Cost That Compounds
Inference is the opposite: almost invisible per event, relentless in aggregate. A single query to a large language model uses a small amount of energy — the calculator on this site takes a central estimate of about 0.3 Wh per query (Epoch AI, 2025), roughly the energy of a modern web search, with a sensitivity band up to around 3.0 Wh for heavier models and longer answers. Nobody notices one query. But a widely used model handles hundreds of millions to billions of them a day.
Inference energy is simply per-query energy × number of queries, and the second term is enormous and growing. At 0.3 Wh a query, a billion daily queries is 300 MWh every day — so in under a week of operation, a popular model’s inference can match the entire one-off energy of training it. The per-query figure feels negligible precisely because the multiplication is hidden; the footprint lives in the volume, not the single event.
The Crossover: Why Inference Usually Wins
Put the two together and the result is counterintuitive: the phase that makes the headlines is usually the smaller one over a model’s life. The crossover is straightforward arithmetic:
- Training energy (GPT-3, reference): ≈ 1,287 MWh = 1,287,000 kWh
- Inference energy: ≈ 0.3 Wh = 0.0003 kWh per query
- Break-even: 1,287,000 ÷ 0.0003 ≈ 4.3 billion queries
After roughly four billion queries, the model has spent as much energy answering questions as it ever spent being trained — and it keeps going. For a model serving hundreds of millions of queries a day, that point arrives in weeks, and everything after it is inference piling far past the training cost. Over a popular model’s service life, inference typically dwarfs training. (Reference figures: Patterson et al. 2021; Epoch AI 2025. Illustrative of the crossover, not a specific product.)
To turn either phase’s energy into emissions, you multiply by the grid’s carbon intensity — today around 0.131 kg CO₂e per kWh on the UK grid, and very different elsewhere. That is why where and when compute runs matters as much as how much: the same queries on a coal-heavy grid emit several times what they would on a clean one.
What It Means for Cutting AI Carbon
The training-versus-inference split reframes where the leverage is. Because inference usually dominates lifetime emissions, the largest reductions for a deployed, widely used model come from making inference cheaper and cleaner — not from agonising over the one-off training bill:
- Inference efficiency at scale. Smaller or distilled models, quantisation, caching, and batching cut the per-query energy — and because it is multiplied by billions, a small per-query saving is a large absolute one.
- Right-sizing the model. Using a model no larger than the task needs is often the biggest inference lever; a frontier model answering trivial queries is pure waste at scale.
- Clean, well-sited compute. Running inference in low-carbon regions and efficient (low-PUE) data centres cuts the grid multiplier on every query.
- Training still counts — once. Efficient training and clean grids matter for the training spike, but it is paid once; do not let it crowd out the larger, ongoing inference question.
The AI compute calculator models both phases with the Patterson reference runs and the per-query band; the AI search calculator focuses on the inference side at query scale.
Common Confusions
- Quoting training carbon as “AI’s footprint”. Training is the visible spike, but for a deployed model inference usually dominates lifetime emissions. Both are needed to describe the footprint.
- Dismissing inference because it is tiny per query. The footprint lives in the volume: a fraction of a watt-hour times billions of queries is a large number.
- Ignoring the grid and PUE. Energy becomes carbon only through the grid factor, and data-centre PUE adds overhead. The same compute has very different carbon in different places.
- Comparing per-query estimates without their assumptions. Inference figures vary widely with model size, output length, and hardware — a central estimate and a sensitivity band, not a single “true” number.
- Forgetting the hardware itself. The accelerators have embodied carbon and become e-waste; compute energy is not the whole AI footprint.
Frequently Asked Questions
Training is the process of building an AI model — feeding it large amounts of data and adjusting its parameters over days or weeks on big clusters of accelerators. It happens once (or occasionally, on retraining) and is very energy-intensive while it runs. Inference is the process of using the finished model: each time it answers a prompt or returns a result, it performs a much smaller burst of computation. The key difference is shape — training is a single tall spike of energy, while inference is a long, low plateau that lasts as long as the model is in service. That is why the two carry carbon so differently, and why an AI footprint needs both to be described honestly. The split parallels embodied versus operational carbon.
For a widely used model, inference usually uses more energy over the model’s life, despite each query being tiny. The reason is volume. Training a large model might take on the order of 1,287 MWh (the reference figure for GPT-3, Patterson et al. 2021), while a single inference query uses roughly 0.3 Wh (Epoch AI 2025). Dividing one by the other, cumulative inference energy overtakes the entire training cost after about four billion queries — a threshold a popular model crosses in weeks, after which inference keeps accumulating. So while training is the larger single event, inference typically dominates lifetime emissions. This is exactly why the biggest carbon lever for a deployed model is usually inference efficiency at scale.
A single query to a large language model uses a small amount of energy — a central estimate of about 0.3 Wh (Epoch AI 2025), roughly comparable to a modern web search, with a sensitivity band up to around 3.0 Wh for larger models and longer answers. To convert that energy to carbon, you multiply by the grid’s carbon intensity — around 0.131 kg CO₂e per kWh on the UK grid, and different elsewhere — so 0.3 Wh works out to a fraction of a gram of CO₂e per query on a relatively clean grid. The number is small per query but should always be read with its assumptions (model size, output length, hardware, and grid), and it compounds enormously across billions of queries.
Because inference energy is per-query energy multiplied by query volume, the biggest reductions come from cutting the per-query cost and cleaning the compute. Using a smaller or distilled model sized to the task, along with quantisation, caching, and batching, cuts the energy of each query — and since it is multiplied by billions, a small per-query saving is a large absolute one. Right-sizing matters most: a frontier model answering trivial queries wastes energy at scale. Running inference in low-carbon regions and efficient, low-PUE data centres reduces the grid multiplier on every query. Training efficiency and clean grids matter too, but training is paid once, whereas inference savings recur on every future query.
It is a close analogy, and a useful one. Training is like embodied carbon: a one-time, front-loaded cost spent to create the asset, fixed the moment it is done. Inference is like operational carbon: spread over the asset’s working life and scaling with how much it is used. Just as an efficient building can flip the balance so that embodied carbon dominates, a rarely used model can be training-dominated — but for a popular model, as for an energy-hungry building, the operational side (inference) usually dominates. The analogy also carries the same lesson: you have to count both phases over the full life to see the real footprint, rather than fixating on the visible upfront number. One caveat: the AI hardware itself also has genuine embodied carbon and becomes e-waste, which sits outside the compute-energy picture entirely.