Sustainability of language models

Summary

A full life-cycle assessment of the sustainability of language models, using a small language model as a measurable proxy for large-scale systems. The work spans environmental, economic, and social impact — from a single training run to the aggregate footprint of a global fleet.

Language models are being woven into global digital infrastructure faster than their costs are understood. The brief was to quantify those costs across the full life cycle: not to estimate them in the abstract, but to measure a real model end to end and reason from there to production scale.

We framed the work as a life-cycle assessment with defined functional units — one training run and one inference — and drew the system boundary around the operational phase, where energy use concentrates, excluding hardware manufacture and decommissioning. The object of study was a character-level, decoder-only GPT-style transformer of roughly 6.5 million parameters, trained on the Tiny Shakespeare dataset on a single laptop.

Energy and power were measured directly with CodeCarbon on an NVIDIA RTX 3060, and emissions derived from the Danish DK2 electricity mix — a weighted average of about 0.059 kg CO₂e per kWh, dominated by wind and low-carbon imports.

From a baseline configuration we ran five single-parameter scenarios, isolating the parameters that drive impact: training iterations, embedding dimension, and batch size for training, and prompt length and generated tokens for inference. Each run was costed against AWS EC2 on-demand pricing as a cloud reference, and translated into human-health terms through a disability-adjusted-life-year impact factor.

To judge whether the impact was “safe”, we placed it inside a planetary boundary: a grandfathering allocation of the 1.5 °C carbon budget that attributes one percent to AI and assumes a million global training runs a year, giving a boundary of roughly 330 kg CO₂e per run. We then compared the small model against a production-scale system using EcoLogits data for Mistral Large 2.

Training dominates the life cycle. A single training run emitted about 0.15 g CO₂e — some 45,000 times a single inference at 0.0000033 g — and scaling the architecture or training duration raised both emissions and cloud cost in near proportion. Every scenario sat several orders of magnitude below the 330 kg boundary.

The headline finding is one of scale. For a single model the impact is negligible; across a global fleet, millions of concurrent deployments compound local compute into real environmental and health burdens that fall hardest on communities near power generation and hardware mining. The practical levers are right-sizing architectures to the task, scheduling heavy training on low-carbon grids, and cutting wasted training cycles — and, longer term, narrowing, slowing, and closing the resource loop through specialized models, open weights, and hardware recycling. All of it set against the rebound effect, where cheaper inference drives enough new demand to erase the per-prompt savings.