Measuring What a Model Costs: Energy as an Evaluation Metric

Whether a model fits on a device is one question about cost. How much energy it takes to keep it running is another.

---

Most efficiency claims in machine learning papers rest on one of three numbers: parameter count, FLOPs, or wall-clock latency on some unspecified machine. I have used all three. This post is about why I now think measured energy, under a stated deployment scenario, is the number that should sit next to accuracy, and about how easy it is to measure it badly.

Where I started: a parameter budget#

In Green-NAS we ran a multi-objective architecture search (NSGA-II) for hourly weather forecasting across 24 cities, about 1.07 million samples in all. The search had three objectives: minimise validation RMSE, minimise parameter count, and prefer shallower networks as a rough proxy for interpretability. With 20 individuals over 10 generations it found a Pareto front of 20 architectures.

The results looked good on the efficiency axis:

Model	Params	RMSE	Inference (ms)
Manual GRU	153K	0.1004	0.29
Manual Transformer	135K	0.0974	1.89
Green-NAS-A (2-layer GRU)	153K	0.0988	0.47
Green-NAS-B	4.2K	0.0996	0.29
Green-NAS-C (CNN)	1.1K	0.1019	0.33

Green-NAS-A came within 1.4% of the best hand-built baseline, and Green-NAS-C got within about 5% of it with roughly a thousand parameters and a 4 KB file. For a microcontroller in a weather station, that last row is the interesting one.

Parameter count was the right target there. On a microcontroller, memory is the binding constraint, and the question is whether the model fits at all. The next question is different: once a model fits, what does it cost to keep it running? That depends on how often it is retrained, what hardware it runs on, and how many forecasts each training serves. Answering it needs a meter, not a parameter count, and that is what I took into the next project.

Measuring energy instead#

In the next project I compared a zero-shot time-series foundation model (Chronos-Bolt) with a tuned LightGBM specialist and a small searched GRU on hourly PM2.5 and temperature forecasting in 29 cities. This time I metered energy directly.

How it was measured matters more than the headline, so here it is:

• What was metered: each model’s entire runner call, wrapped with codecarbon at one-second sampling. That includes loading the checkpoint and, for trained models, the training, not only inference.
• GPU energy came from NVML hardware counters on an RTX 3060 Ti.
• CPU energy did not come from hardware. The machine runs Windows, where the RAPL interface is unavailable, so codecarbon estimates CPU draw from a thermal-design-power model scaled by utilisation. I treat the CPU numbers as directional comparisons only.
• Repeatability: cells that failed a repeatability check (4 of 15) are reported as ranges, not points.
• The deployment scenario: each trained model was charged one full training per 144 hourly forecasts, which works out to retraining every six days for a series forecast 24 hours ahead once a day.

That last line is the assumption the whole result depends on.

The result, and its condition#

On PM2.5 the accuracy question came out as a tie: panel-mean MASE of 0.662 for zero-shot Chronos-Bolt against 0.692 for the tuned specialist under deployable covariates, a difference that was not statistically significant.

On energy, under the six-day retraining protocol, the specialist used 9.5 to 15.1 kJ per 1,000 forecasts across the three replication cities. Chronos-Bolt used 1.0 to 1.2 kJ on the consumer GPU. The searched GRU, which also has to be trained locally, sat with the specialist at 7.3 to 16.8 kJ. That is about an order of magnitude in energy and about 8× in electricity cost.

A single-city check suggested model size is not what drives this. On Beijing, TimesFM (4.8× more parameters than Chronos-Bolt) used 1.15 kJ per 1,000 forecasts against 0.53 kJ for Chronos-Bolt. Both were still below the 3.22 kJ the retrained specialist needed on the same series. The specialist’s cost is mostly retraining, not inference.

The obvious next question: what if you train once and never retrain?

When I separated one-time training energy from per-forecast inference energy, the picture reversed. At inference alone, the specialist’s 24 small per-horizon tree models are cheaper per forecast than one Chronos-Bolt forward pass. A specialist trained once and then reused overtakes the foundation model after about 2,800 to 4,400 forecasts on CPU, roughly 115 to 185 days of daily forecasting for one series, and after 250 to 2,100 forecasts on GPU.

So the honest statement is conditional. The zero-shot model’s energy advantage holds for deployments that retrain more often than every few months. It is not a general property of foundation models.

Keeping the number in proportion#

One more figure puts all of this in context. Both options cost less than 10⁻³ USD in electricity per 1,000 forecasts (at 0.15 USD/kWh and a data-centre overhead factor of 1.4). In absolute money, the energy difference is small next to hardware, memory and engineering time.

I think that is worth saying plainly, because “10× less energy” reads as a bigger deal than it is for a single city. Where the difference starts to matter is scale: thousands of monitoring stations, each needing its own specialist retrained on a schedule, against one downloaded checkpoint. The practical advantage of the zero-shot model is mostly that it skips the training step, and with it a whole pipeline someone has to maintain.

What I’d recommend reporting#

If you want to make an efficiency claim, these are the things I now try to include:

1. State the deployment scenario. How often is the model retrained? How many forecasts does each training serve? Without this, energy numbers cannot be compared.
2. Split training from inference. Then give the break-even point. A reader can plug in their own retraining schedule.
3. Say how energy was measured. Hardware counters or a power model? On which device? Which part of the pipeline was metered?
4. Report ranges when repeatability fails, rather than a mean that hides the spread.
5. Keep parameter count as a secondary figure. It still matters for memory and for fitting on a device. It is not a cost.
6. Give absolute cost as well as ratios. A ratio can make a small absolute difference look dramatic.

Limits#

• Energy was measured on one consumer machine. CPU figures come from a power model, so they will not transfer exactly to other processors.
• The panel energy comparison used three replication cities, and the TimesFM check was a single city.

Accuracy tables are everywhere in this field. Cost tables are rare, and when they appear they usually count something convenient. Metered energy is harder to get and full of caveats, but at least the caveats are visible.

The foundation-model study is summarised on its publication page (preprint at Research Square), and Green-NAS is on arXiv.