Method
How we estimate
A Measured value is always preferred. Otherwise LocalFit estimates, shown with “~”. Calibration is recomputed at every build from 916 decode and 1,041 prefill observations that are single-device, resolved to a catalogued GPU/Mac, free of research review blockers, at shallow context and with a known quant width.
Generation and prompt speed
- Bytes per token = the actual file size (dense) or file size × active ÷ total parameters (MoE). KV reads are ignored (short context).
- If a device (or the same Apple chip and GPU-core count) has ≥3 measured points on the backend, its median effective bandwidth / FLOP rate is used instead, which also covers devices without published figures.
- Apple prefill otherwise uses the median measured prefill FLOP rate per GPU core of the same chip generation (e.g. all M4 variants, ≥5 points from ≥2 chips) × the chip's GPU-core count, because Apple publishes no GPU FP16 figure.
- Backends: CUDA for NVIDIA, ROCm then Vulkan for AMD GPUs, Vulkan for AMD unified memory, Metal or MLX for Macs, SYCL/Vulkan for Intel. With no calibration for the backend, the value stays Unknown.
Decode efficiency tokens/s × bytes per token ÷ bandwidth
| Backend | Model | Median efficiency | Points | Devices | Leave-one-device-out error | |
|---|---|---|---|---|---|---|
| llama.cpp CUDA | Dense | 0.6735 | 179 | 21 | 12% | used |
| llama.cpp CUDA | MoE | 0.4832 | 46 | 11 | 18% | used |
| llama.cpp Metal | Dense | 0.7459 | 185 | 31 | 11% | used |
| llama.cpp Metal | MoE | 0.2602 | 4 | 2 | — | too few points |
| MLX | Dense | 0.8231 | 60 | 18 | 10% | used |
| MLX | MoE | 0.3416 | 94 | 20 | 30% | used |
| llama.cpp ROCm | Dense | 0.5586 | 117 | 9 | 10% | used |
| llama.cpp ROCm | MoE | 0.3767 | 2 | 1 | — | too few points |
| llama.cpp Vulkan · AMD | Dense | 0.6706 | 129 | 8 | 15% | used |
| llama.cpp Vulkan · AMD | MoE | 0.5174 | 8 | 4 | 73% | used |
| llama.cpp Vulkan · NVIDIA | Dense | 0.7063 | 68 | 15 | 10% | used |
| llama.cpp Vulkan · NVIDIA | MoE | 0.7692 | 6 | 6 | 32% | used |
Prefill efficiency tokens/s × 2 × active params ÷ FP16 FLOP/s
| Backend | Model | Median efficiency | Points | Devices | Leave-one-device-out error | |
|---|---|---|---|---|---|---|
| llama.cpp CUDA | Dense | 0.9489 | 149 | 13 | 18% | used |
| llama.cpp CUDA | MoE | 0.4456 | 24 | 8 | 25% | used |
| llama.cpp ROCm | Dense | 0.3468 | 116 | 9 | 19% | used |
| llama.cpp ROCm | MoE | 0.1047 | 7 | 2 | — | used |
| llama.cpp Vulkan · AMD | Dense | 0.285 | 141 | 8 | 28% | used |
| llama.cpp Vulkan · AMD | MoE | 0.08279 | 23 | 5 | 75% | used |
| llama.cpp Vulkan · NVIDIA | Dense | 0.7918 | 59 | 10 | 18% | used |
| llama.cpp Vulkan · NVIDIA | MoE | 0.1815 | 7 | 4 | 44% | used |
Error = median absolute error of each device's efficiency when predicted from the other devices. MoE and low-sample groups are much less reliable. Apple prefill relies on device-level calibration because Apple does not publish GPU FP16 throughput.
First token
It excludes model load, queueing and any reasoning phase before the first visible token. Only 8 explicitly measured end-to-end TTFT observations exist in the database; they are shown as Measured only for their own test.
Unknown values
Unknown means no source and no defensible estimate: for example an unpublished FP16 or bandwidth figure with no device calibration, a MoE without a published active size, or a cache formula that the research could not establish. Unknown is never treated as zero; memory totals with unknown terms are marked “conditional” or “+”.
Not estimated
Power draw and GPU utilisation are not shown: vendor power ratings are not measured LLM load, and the earlier heuristic was not calibrated.