Tokens per watt: how efficient is your local AI box?

Tokens per watt is really tokens per joule: divide output tokens per second by the watts your box draws at the wall. Multiply by 3,600,000 to get tokens per kWh. Example: 30 tok/s at 350 W is about 309,000 tokens per kWh, or roughly €0.97 of electricity per million tokens at €0.30/kWh.

The short answer

Tokens per joule = output tokens per second ÷ watts at the wall. A watt is one joule per second, so the seconds cancel. Multiply by 3,600,000 (the joules in a kWh) and you have tokens per kWh. From there, the electricity cost of a million tokens is:

cost per 1M tokens = 1,000,000 ÷ tokens per kWh × your price per kWh

Tokens per second tells you how fast a box is. Tokens per kWh tells you what it costs to run.

The calculator

Enter output tokens per second, watts at the wall while generating, and your price per kWh. Add idle watts and hours of generation per day to see the all-in number for a box that sits idle most of the day.

Input	Default
Output tokens per second	30
Watts at the wall while generating	350
Price per kWh	0.30 (€ / $ / £ picker)
Idle watts (optional)	90
Hours generating per day (optional)	2

Output	With the defaults
Tokens per joule	0.086
Joules per token	11.7
Tokens per kWh	≈ 309,000
Electricity per 1M tokens	≈ €0.97
All-in per day (2 h generating, 22 h idle)	2.68 kWh, 216,000 tokens, ≈ €0.80
All-in tokens per kWh	≈ 80,600
All-in electricity per 1M tokens	≈ €3.72

The last three rows are the point of the optional inputs. A box that generates for two hours and idles for twenty-two spends most of its energy doing nothing, so the cost per million tokens you actually produce is several times the “while generating” figure.

How to measure the two inputs

Tokens per second (output, generation only).

• Ollama: ollama run <model> --verbose prints the eval rate after each answer. Through the API, the response carries eval_count (tokens generated) and eval_duration (nanoseconds): tok/s = eval_count ÷ eval_duration × 10⁹.
• Use a realistic prompt and a reasonably filled context. Prompt processing (prefill) is a different, much faster number; don’t mix the two.
• Average several runs over a few minutes, not one short answer.

Watts at the wall.

• A smart plug or a plug-in meter on the AI box, read while it generates. This is the number your bill sees.
• Your UPS, if the box is the only thing on it: many UPSes report output watts, or a load percentage you can multiply by the rated watts.
• The GPU’s own reading (nvidia-smi --query-gpu=power.draw --format=csv -l 1, or powermetrics on a Mac) leaves out the CPU, RAM, fans and power-supply losses. It’s useful for comparing settings, not for cost.

What moves the number

• Model size and quantisation. A 2026 preprint that measured 1B–7B models on a 16 GB consumer GPU found the smallest models used under 0.7 joules per token, and a 7B model up to 4.4× more energy per token than the most efficient one. Architecture and quantisation mattered as much as parameter count.
• Power limits. Generation is mostly memory-bound, so a moderate GPU power limit often costs little speed. One public test cut a card from 180 W to 150 W, kept essentially the same tokens per second, and gained about 16 % efficiency.
• Reasoning-style models. Tokens per kWh can look fine while energy per answer is high, because the model writes many hidden tokens before replying. If you care about cost per answer, measure that too.
• Idle. The biggest lever for a home box that isn’t busy all day: the all-in rows above. Spinning down idle GPUs, or putting the model on a machine that idles low, can matter more than any tuning.

Share your numbers

We’re compiling a community table of real measurements. Post yours in this format (r/LocalLLaMA thread, or e-mail da@mustardseedmt.com), credited by username or anonymous:

GPU or box · runtime · model size + quant · output tok/s · watts at the wall · idle watts

The table appears here once at least ten measurements are in.

Sources

• arXiv preprint 2608.00008: energy per token of small language models on a consumer GPU (2026)
• Ollama API documentation: eval_count and eval_duration in generate responses
• InventiveHQ: GPU power limit versus tokens per watt test

Questions

What is a good tokens per kWh for a home AI box?

It depends mostly on model size and quantisation. Small models on a modern GPU reach well over a million tokens per kWh; large models on a single consumer GPU can land in the low hundreds of thousands. Compare your own box before and after a change rather than against someone else's.

Should I use the GPU's own power reading or a wall meter?

For cost, use the wall: that is what your bill counts, and it includes the CPU, RAM, fans and power-supply losses that the GPU reading leaves out. The GPU reading is fine for comparing two GPUs or two settings.

How do I get tokens per second from Ollama?

Run a prompt with ollama run --verbose and read the eval rate, or use the API: the response includes eval_count (tokens generated) and eval_duration (nanoseconds), so tokens per second = eval_count ÷ eval_duration × 1,000,000,000.

Does a lower GPU power limit hurt speed?

Often less than you'd expect. Generating tokens is mostly limited by memory bandwidth, so a moderate power limit can keep tokens per second almost the same while raising tokens per kWh. Measure both before and after.