NVIDIA DGX Spark (128 GB): which models fit?

RTX 4090 RTX 5090 RTX 3090 RTX 4060 Ti (16 GB)RTX 5070 Ti Radeon RX 7900 XTX M4 Max 128 GB M3 Ultra 512 GB DGX Spark (128 GB)All devices

NVIDIA · Unified-memory PC

NVIDIA DGX Spark (128 GB)

Memory128 GB LPDDR5XBandwidth273 GB/s vendor specPlanned for models96.0 GiB 75% of unified memoryFilesGGUF llama.cpp format

Budget = file + KV cache + 1 GiB runtime + 0.5 GiB spare. Macs: 75% of memory. Details

43 models fit DGX Spark (128 GB) · 8K context

Ranked by Artificial Analysis Intelligence Index, unranked models last. Arena is shown separately. How ranking works

Intelligence

Artificial Analysis Intelligence Index · top 10 models that fit DGX Spark (128 GB) · higher is better

AA estimate

Generation speed

Tokens per second on DGX Spark (128 GB), best file for each of the same models · higher is better

MeasuredEstimated (~)

WeightsKV cacheRuntimeFreeMeasuredlinks to its sourceEstimatedshown with ~Unknownnever zero

Qwen3.8-27B

Qwen/Qwen3.8-27B

2026-08-1427B dense

AA index33.7rank #13

Arena1,441

Best file & memory

Q8_027.1 GiB fileFits · conditional

28.6 of 96.0 GiB+8K ctx · max ≈256K

Prompt~826 t/sEstimated

Generate~7.6 t/sEstimated

First token · 4K prompt~5.0 sEstimated

Memory budget

Weights file	27.12 GiB	bartowski/Qwen3.8-27B-GGUF HF files
KV cache · 8K	0.50 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	28.62 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈256K tokens (native 256K).

Other GGUF files

Q4_K_M 16.2 GiB · fits · files
Q5_K_M 19.5 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 33.7 Qwen3.8 27B (Xhigh) · Arena 1,441 · Model page

G9v3-39A5B (preview)

ai9stars/G9v3-39A5B

2026-08 (day unknown)39B · 5B active

AA index21.8 AA estimaterank #29

Arena—

Best file & memory

Q8_038.6 GiB fileFits

39.9 of 96.0 GiB8K ctx · max ≈128K

Prompt~1,913 t/sEstimated

Generate~29.7 t/sEstimated

First token · 4K prompt~2.1 sEstimated

Memory budget

Weights file	38.63 GiB	linuxid10t/G9v3-39A5B-GGUF HF files
KV cache · 8K	0.30 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	39.93 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈128K tokens (native 128K).

Other GGUF files

Q4_K_M 21.9 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 21.8 AA estimate G9v3-39A5B · Arena unknown · Model page

NVIDIA Nemotron 3.5 Bolt 74B-A9B

nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16

2026-10-1574B · 9B active

AA index21.4rank #32

Arena—

Best file & memory

Q8_073.4 GiB fileFits · conditional

74.5 of 96.0 GiB+8K ctx · max ≈256K

Prompt~1,063 t/sEstimated

Generate~16.5 t/sEstimated

First token · 4K prompt~3.9 sEstimated

Memory budget

Weights file	73.44 GiB	unsloth/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-GGUF · 3 files HF files
KV cache · 8K	0.06 GiB	BF16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	74.50 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈256K tokens (native 256K).

Other GGUF files

BF16 138 GiB · too big · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Sliding-window allocation is not established.
Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 21.4 Nemotron 3.5 Bolt 74B-A9B · Arena unknown · Model page

K2-Horizon-7B

IFM/K2-Horizon-7B

2026-09-03size unknown

AA index20.6rank #33

Arena—

Best file & memory

Q8_08.9 GiB fileFits

11.0 of 96.0 GiB8K ctx · max ≈512K

Prompt—Unknown

Generate~23 t/sEstimated

First token · 4K prompt—Unknown

Memory budget

Weights file	8.92 GiB	IFM/K2-Horizon-7B-GGUF HF files
KV cache · 8K	1.13 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	11.04 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈512K tokens (native 512K).

Other GGUF files

Q4_K_M 5.2 GiB · fits · files
Q5_K_M 6.0 GiB · fits · files
FP16 16.8 GiB · fits · files

Where the speeds come from

Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token unknown: needs a prefill value.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 20.6 K2 Horizon 7B · Arena unknown · Model page

Qwen3.6-35B-A3B

Qwen/Qwen3.6-35B-A3B

2026-04-1635B · 3B active

AA index18.2rank #36

Arena—

Best file & memory

Q8_034.4 GiB fileFits · conditional

35.5 of 96.0 GiB+8K ctx · max ≈256K

Prompt~3,188 t/sEstimated

Generate~50 t/sEstimated

First token · 4K prompt~1.3 sEstimated

Memory budget

Weights file	34.37 GiB	unsloth/Qwen3.6-35B-A3B-GGUF HF files
KV cache · 8K	0.16 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	35.52 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈256K tokens (native 256K).

Other GGUF files

Q4_K_M 19.0 GiB · fits · files
Q5_K_M 23.0 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 18.2 Qwen3.6 35B A3B (Reasoning) · Arena unknown · Model page

Qwen3.5-122B-A10B

Qwen/Qwen3.5-122B-A10B

2026-02-24122B · 10B active

AA index17.7 AA estimaterank #37

Arena1,417

Best file & memory

Q5_K_M85.2 GiB fileFits · conditional

86.4 of 96.0 GiB+8K ctx · max ≈256K

Prompt~957 t/sEstimated

Generate~21.1 t/sEstimated

First token · 4K prompt~4.3 sEstimated

Memory budget

Weights file	85.23 GiB	unsloth/Qwen3.5-122B-A10B-GGUF · 3 files HF files
KV cache · 8K	0.19 GiB	BF16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	86.42 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈256K tokens (native 256K).

Other GGUF files

Q4_K_M 71.3 GiB · fits · files
Q8_0 121 GiB · too big · files
BF16 228 GiB · too big · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 17.7 AA estimate Qwen3.5 122B A10B (Non-reasoning) · Arena 1,417 · Model page

Muse Glimmer 30B

meta-models/Muse-Glimmer-30B

2026-08-1030B dense

AA index17.5rank #38

Arena1,389

Best file & memory

Q8_027.6 GiB fileFits

28.8 of 96.0 GiB8K ctx · max ≈128K

Prompt~754 t/sEstimated

Generate~7.4 t/sEstimated

First token · 4K prompt~5.4 sEstimated

Memory budget

Weights file	27.58 GiB	bartowski/Muse-Glimmer-30B-GGUF HF files
KV cache · 8K	0.18 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	28.76 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈128K tokens (native 128K).

Other GGUF files

Q4_K_M 15.6 GiB · fits · files
Q5_K_M 18.7 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 17.5 Muse Glimmer (High) · Arena 1,389 · Model page

K2-Horizon-3.7B

IFM/K2-Horizon-3.7B

2026-09-03size unknown

AA index15.6rank #41

Arena—

Best file & memory

Q8_05.0 GiB fileFits

7.1 of 96.0 GiB8K ctx · max ≈512K

Prompt—Unknown

Generate~40.9 t/sEstimated

First token · 4K prompt—Unknown

Memory budget

Weights file	5.02 GiB	IFM/K2-Horizon-3.7B-GGUF HF files
KV cache · 8K	1.13 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	7.14 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈512K tokens (native 512K).

Other GGUF files

Q4_K_M 2.9 GiB · fits · files
Q5_K_M 3.4 GiB · fits · files
FP16 9.4 GiB · fits · files

Where the speeds come from

Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token unknown: needs a prefill value.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 15.6 K2 Horizon 3.7B · Arena unknown · Model page

Mistral Medium 3.5 (128B dense)

mistralai/Mistral-Medium-3.5-128B

2026-04-28128B dense

AA index14.2rank #43

Arena1,421

Best file & memory

Q5_K_M82.3 GiB fileFits · conditional

86.0 of 96.0 GiB+8K ctx · max ≈36K

Prompt~174 t/sEstimated

Generate~2.5 t/sEstimated

First token · 4K prompt~24 sEstimated

Memory budget

Weights file	82.25 GiB	unsloth/Mistral-Medium-3.5-128B-GGUF · 3 files HF files
KV cache · 8K	2.75 GiB	BF16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	86.00 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈36K tokens (native 256K).

Other GGUF files

Q4_K_M 69.8 GiB · fits · files
Q8_0 124 GiB · too big · files
BF16 233 GiB · too big · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 14.2 Mistral Medium 3.5 · Arena 1,421 · Model page

Qwen3.5-9B

Qwen/Qwen3.5-9B

2026-03-029B dense

AA index13.3 AA estimaterank #45

Arena—

Best file & memory

Q8_08.9 GiB fileFits · conditional

10.1 of 96.0 GiB+8K ctx · max ≈256K

Prompt~2,478 t/sEstimated

Generate~23.1 t/sEstimated

First token · 4K prompt~1.7 sEstimated

Memory budget

Weights file	8.87 GiB	unsloth/Qwen3.5-9B-GGUF HF files
KV cache · 8K	0.25 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	10.12 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈256K tokens (native 256K).

Other GGUF files

Q4_K_M 5.3 GiB · fits · files
Q5_K_M 6.1 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 13.3 AA estimate Qwen3.5 9B (Non-reasoning) · Arena unknown · Model page

EXAONE 4.5 33B

LGAI-EXAONE/EXAONE-4.5-33B

2026-04-0933B dense

AA index13.2 AA estimaterank #46

Arena—

Best file & memory

Q8_032.7 GiB fileFits

35.0 of 96.0 GiB8K ctx · max ≈256K

Prompt~676 t/sEstimated

Generate~6.3 t/sEstimated

First token · 4K prompt~6.1 sEstimated

Memory budget

Weights file	32.73 GiB	LGAI-EXAONE/EXAONE-4.5-33B-GGUF HF files
KV cache · 8K	1.25 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	34.98 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈256K tokens (native 256K).

Other GGUF files

Q4_K_M 18.7 GiB · fits · files
Q5_K_M 21.9 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 13.2 AA estimate EXAONE 4.5 33B (Reasoning) · Arena unknown · Model page

Qwen3.5-4B

Qwen/Qwen3.5-4B

2026-03-024B dense

AA index13.1 AA estimaterank #48

Arena—

Best file & memory

Q8_04.2 GiB fileFits · conditional

5.4 of 96.0 GiB+8K ctx · max ≈256K

Prompt~5,576 t/sEstimated

Generate~49.2 t/sEstimated

First token · 4K prompt~735 msEstimated

Memory budget

Weights file	4.17 GiB	unsloth/Qwen3.5-4B-GGUF HF files
KV cache · 8K	0.25 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	5.42 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈256K tokens (native 256K).

Other GGUF files

Q4_K_M 2.6 GiB · fits · files
Q5_K_M 2.9 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 13.1 AA estimate Qwen3.5 4B (Reasoning) · Arena unknown · Model page

NVIDIA Nemotron 3.5 Lightning 30B-A3B

nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

2026-08-1130B · 3B active

AA index12.9rank #49

Arena1,329

Best file & memory

Q8_032.6 GiB fileFits · conditional

33.6 of 96.0 GiB+8K ctx · max ≈256K

Prompt~3,188 t/sEstimated

Generate~45.2 t/sEstimated

First token · 4K prompt~1.3 sEstimated

Memory budget

Weights file	32.60 GiB	bartowski/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF HF files
KV cache · 8K	0.05 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	33.65 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈256K tokens (native 256K).

Other GGUF files

Q4_K_M 23.7 GiB · fits · files
Q5_K_M 25.1 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 12.9 Nemotron 3.5 Lightning · Arena 1,329 · Model page

IBM Granite 4.2 30B

ibm-granite/granite-4.2-30b

2026-08-2530B dense

AA index12.7rank #50

Arena1,361

Best file & memory

Q8_029.0 GiB fileFits

32.0 of 96.0 GiB8K ctx · max ≈128K

Prompt~744 t/sEstimated

Generate~7.1 t/sEstimated

First token · 4K prompt~5.5 sEstimated

Memory budget

Weights file	28.98 GiB	ibm-granite/granite-4.2-30b-GGUF HF files
KV cache · 8K	2.00 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	31.98 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈128K tokens (native 128K).

Other GGUF files

Q4_K_M 16.5 GiB · fits · files
Q5_K_M 19.3 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 12.7 Granite 4.2 30B · Arena 1,361 · Model page

MiniCPM5-2B

openbmb/MiniCPM5-2B

2026-09-072.5B dense

AA index12.5rank #51

Arena—

Best file & memory

Q8_02.5 GiB fileFits

3.8 of 96.0 GiB8K ctx · max ≈128K

Prompt~8,863 t/sEstimated

Generate~82.2 t/sEstimated

First token · 4K prompt~462 msEstimated

Memory budget

Weights file	2.50 GiB	openbmb/MiniCPM5-2B-GGUF HF files
KV cache · 8K	0.33 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	3.82 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈128K tokens (native 128K).

Other GGUF files

Q4_K_M 1.5 GiB · fits · files
Q5_K_M 1.8 GiB · fits · files
FP16 4.7 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 12.5 MiniCPM5-2B · Arena unknown · Model page

HyperNova 60B 2605 (compressed gpt-oss-120b)

MultiverseComputingCAI/Hypernova-60B-2605

2026-05-0660B · 4.8B active

AA index11.7 AA estimaterank #52

Arena—

Best file & memory

Q8_058.1 GiB fileFits · conditional

59.4 of 96.0 GiB+8K ctx · max ≈128K

Prompt~1,993 t/sEstimated

Generate~31.7 t/sEstimated

First token · 4K prompt~2.1 sEstimated

Memory budget

Weights file	58.13 GiB	mradermacher/Hypernova-60B-2605-GGUF HF files
KV cache · 8K	0.25 GiB	BF16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	59.39 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈128K tokens (native 128K).

Other GGUF files

Q4_K_M 37.9 GiB · fits · files
Q5_K_M 41.3 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 11.7 AA estimate HyperNova 60B 2605 (High, Based on gpt-oss-120b) · Arena unknown · Model page

Nemotron Cascade 2 30B A3B

nvidia/Nemotron-Cascade-2-30B-A3B

2026-03-1930B · 3B active

AA index11.7 AA estimaterank #53

Arena—

Best file & memory

Q8_031.3 GiB fileFits · conditional

32.3 of 96.0 GiB+8K ctx · max ≈256K

Prompt~3,188 t/sEstimated

Generate~47.1 t/sEstimated

First token · 4K prompt~1.3 sEstimated

Memory budget

Weights file	31.28 GiB	mradermacher/Nemotron-Cascade-2-30B-A3B-GGUF HF files
KV cache · 8K	0.05 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	32.33 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈256K tokens (native 256K).

Other GGUF files

Q4_K_M 22.8 GiB · fits · files
Q5_K_M 24.3 GiB · fits · files
FP16 58.8 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 11.7 AA estimate Nemotron Cascade 2 30B A3B · Arena unknown · Model page

Granite 4.2 8B

ibm-granite/granite-4.2-8b

2026-08-258B dense

AA index11.1rank #56

Arena1,320

Best file & memory

Q8_08.7 GiB fileFits

11.0 of 96.0 GiB8K ctx · max ≈128K

Prompt~2,788 t/sEstimated

Generate~23.6 t/sEstimated

First token · 4K prompt~1.5 sEstimated

Memory budget

Weights file	8.70 GiB	ibm-granite/granite-4.2-8b-GGUF HF files
KV cache · 8K	1.25 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	10.95 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈128K tokens (native 128K).

Other GGUF files

Q4_K_M 5.0 GiB · fits · files
Q5_K_M 5.8 GiB · fits · files
FP16 16.4 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 11.1 Granite 4.2 8B · Arena 1,320 · Model page

G9v3-3B

AI9Stars/G9v3-3B

2026-07 (day unknown)3B dense

AA index10.8 AA estimaterank #58

Arena—

Best file & memory

Q8_03.0 GiB fileFits

4.4 of 96.0 GiB8K ctx · max ≈128K

Prompt~7,435 t/sEstimated

Generate~69.3 t/sEstimated

First token · 4K prompt~551 msEstimated

Memory budget

Weights file	2.96 GiB	mradermacher/G9v3-3B-GGUF HF files
KV cache · 8K	0.41 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	4.37 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈128K tokens (native 128K).

Other GGUF files

Q4_K_M 1.7 GiB · fits · files
Q5_K_M 2.0 GiB · fits · files
FP16 5.6 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 10.8 AA estimate G9v3-3B · Arena unknown · Model page

North Mini Code

CohereLabs/North-Mini-Code-1.0

2026-06-0930B · 3B active

AA index9.9rank #61

Arena—

Best file & memory

Q8_030.2 GiB fileFits

31.7 of 96.0 GiB8K ctx · max ≈250K

Prompt~3,188 t/sEstimated

Generate~48.8 t/sEstimated

First token · 4K prompt~1.3 sEstimated

Memory budget

Weights file	30.21 GiB	bartowski/North-Mini-Code-1.0-GGUF HF files
KV cache · 8K	0.48 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	31.69 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈250K tokens (native 250K).

Other GGUF files

Q4_K_M 17.5 GiB · fits · files
Q5_K_M 20.3 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 9.9 North Mini Code · Arena unknown · Model page

Qwen3-Coder-Next

Qwen/Qwen3-Coder-Next

2026-02-0280B · 3B active

AA index9.2rank #63

Arena—

Best file & memory

Q8_079.0 GiB fileFits · conditional

80.2 of 96.0 GiB+8K ctx · max ≈256K

Prompt~3,188 t/sEstimated

Generate~49.7 t/sEstimated

First token · 4K prompt~1.3 sEstimated

Memory budget

Weights file	78.99 GiB	unsloth/Qwen3-Coder-Next-GGUF · 3 files HF files
KV cache · 8K	0.19 GiB	BF16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	80.18 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈256K tokens (native 256K).

Other GGUF files

Q4_K_M 45.2 GiB · fits · files
Q5_K_M 52.9 GiB · fits · files
BF16 149 GiB · too big · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 9.2 Qwen3 Coder Next · Arena unknown · Model page

Granite 4.2 3B

ibm-granite/granite-4.2-3b

2026-08-253B dense

AA index9.1rank #64

Arena1,293

Best file & memory

Q8_03.6 GiB fileFits

5.3 of 96.0 GiB8K ctx · max ≈128K

Prompt~7,435 t/sEstimated

Generate~56.6 t/sEstimated

First token · 4K prompt~551 msEstimated

Memory budget

Weights file	3.63 GiB	ibm-granite/granite-4.2-3b-GGUF HF files
KV cache · 8K	0.63 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	5.25 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈128K tokens (native 128K).

Other GGUF files

Q4_K_M 2.1 GiB · fits · files
Q5_K_M 2.4 GiB · fits · files
FP16 6.8 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 9.1 Granite 4.2 3B · Arena 1,293 · Model page

Tri-21B-Think

trillionlabs/Tri-21B-Think

2026-02 (day unknown)21B dense

AA index9.0 AA estimaterank #65

Arena—

Best file & memory

Q8_020.5 GiB fileFits

23.1 of 96.0 GiB8K ctx · max ≈32K

Prompt~1,076 t/sEstimated

Generate~10 t/sEstimated

First token · 4K prompt~3.8 sEstimated

Memory budget

Weights file	20.52 GiB	mykor/Tri-21B-Think-GGUF HF files
KV cache · 8K	1.56 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	23.08 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈32K tokens (native 32K).

Other GGUF files

Q4_K_M 11.7 GiB · fits · files
Q5_K_M 13.7 GiB · fits · files
FP16 38.6 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 9.0 AA estimate Tri-21B-Think · Arena unknown · Model page

MiniCPM5-1B

openbmb/MiniCPM5-1B

2026-05-191.1B dense

AA index8.8 AA estimaterank #67

Arena—

Best file & memory

Q8_01.1 GiB fileFits

2.3 of 96.0 GiB8K ctx · max ≈128K

Prompt~20,641 t/sEstimated

Generate~191 t/sEstimated

First token · 4K prompt~198 msEstimated

Memory budget

Weights file	1.07 GiB	openbmb/MiniCPM5-1B-GGUF HF files
KV cache · 8K	0.19 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	2.26 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈128K tokens (native 128K).

Other GGUF files

Q4_K_M 0.6 GiB · fits · files
Q5_K_M 0.7 GiB · fits · files
FP16 2.0 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 8.8 AA estimate MiniCPM5-1B (Reasoning) · Arena unknown · Model page

Nanbeige4.1-3B

Nanbeige/Nanbeige4.1-3B

2026-02 (day unknown)3B dense

AA index8.4 AA estimaterank #69

Arena—

Best file & memory

Q8_03.9 GiB fileFits

5.4 of 96.0 GiB8K ctx · max ≈256K

Prompt~7,435 t/sEstimated

Generate~52.7 t/sEstimated

First token · 4K prompt~551 msEstimated

Memory budget

Weights file	3.90 GiB	mradermacher/Nanbeige4.1-3B-GGUF HF files
KV cache · 8K	0.50 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	5.40 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈256K tokens (native 256K).

Other GGUF files

Q4_K_M 2.3 GiB · fits · files
Q5_K_M 2.6 GiB · fits · files
FP16 7.3 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 8.4 AA estimate Nanbeige4.1-3B · Arena unknown · Model page

LFM2.5-2.6B

LiquidAI/LFM2.5-2.6B

2026-08-042.7B dense

AA index8.4 AA estimaterank #70

Arena—

Best file & memory

Q8_02.7 GiB fileFits · conditional

3.8 of 96.0 GiB+8K ctx · max ≈128K

Prompt~8,292 t/sEstimated

Generate~76.7 t/sEstimated

First token · 4K prompt~494 msEstimated

Memory budget

Weights file	2.68 GiB	LiquidAI/LFM2.5-2.6B-GGUF HF files
KV cache · 8K	0.13 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	3.80 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈128K tokens (native 128K).

Other GGUF files

Q4_K_M 1.6 GiB · fits · files
Q5_K_M 1.8 GiB · fits · files
FP16 5.0 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 8.4 AA estimate LFM2.5-2.6B · Arena unknown · Model page

Falcon-H1R-7B

tiiuae/Falcon-H1R-7B

2026-01-057B dense

AA index7.8 AA estimaterank #71

Arena—

Best file & memory

Q8_07.5 GiB fileFits · conditional

8.9 of 96.0 GiB+8K ctx · max ≈256K

Prompt~3,186 t/sEstimated

Generate~27.3 t/sEstimated

First token · 4K prompt~1.3 sEstimated

Memory budget

Weights file	7.51 GiB	tiiuae/Falcon-H1R-7B-GGUF HF files
KV cache · 8K	0.34 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	8.86 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈256K tokens (native 256K).

Other GGUF files

Q4_K_M 4.3 GiB · fits · files
Q5_K_M 5.0 GiB · fits · files
FP16 14.1 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 7.8 AA estimate Falcon-H1R-7B · Arena unknown · Model page

Step3 VL 10B

stepfun-ai/Step3-VL-10B

2026-01 (day unknown)10B dense

AA index7.7 AA estimaterank #73

Arena—

Best file & memory

Q8_08.1 GiB fileFits

10.2 of 96.0 GiB8K ctx · max ≈64K

Prompt~2,231 t/sEstimated

Generate~25.3 t/sEstimated

First token · 4K prompt~1.8 sEstimated

Memory budget

Weights file	8.11 GiB	JamePeng2023/Step3-VL-10B-GGUF HF files
KV cache · 8K	1.13 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	10.24 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈64K tokens (native 64K).

Other GGUF files

Q4_K_M 4.7 GiB · fits · files
Q5_K_M 5.4 GiB · fits · files
FP16 15.3 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 7.7 AA estimate Step3 VL 10B · Arena unknown · Model page

LFM2.5-8B-A1B

LiquidAI/LFM2.5-8B-A1B

2026-05-288.3B · 1.5B active

AA index7.2 AA estimaterank #74

Arena—

Best file & memory

Q8_08.4 GiB fileFits · conditional

9.5 of 96.0 GiB+8K ctx · max ≈125K

Prompt~6,377 t/sEstimated

Generate~97.2 t/sEstimated

First token · 4K prompt~642 msEstimated

Memory budget

Weights file	8.39 GiB	LiquidAI/LFM2.5-8B-A1B-GGUF HF files
KV cache · 8K	0.09 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	9.49 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈125K tokens (native 125K).

Other GGUF files

Q4_K_M 4.8 GiB · fits · files
Q5_K_M 5.6 GiB · fits · files
FP16 15.8 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 7.2 AA estimate LFM2.5-8B-A1B · Arena unknown · Model page

NVIDIA Nemotron 3 Nano 4B

nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16

2026-03-164B dense

AA index4.8rank #75

Arena—

Best file & memory

Q8_03.9 GiB fileFits · conditional

5.1 of 96.0 GiB+8K ctx · max ≈256K

Prompt~5,618 t/sEstimated

Generate~52.1 t/sEstimated

First token · 4K prompt~729 msEstimated

Memory budget

Weights file	3.94 GiB	unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF HF files
KV cache · 8K	0.13 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	5.07 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈256K tokens (native 256K).

Other GGUF files

Q4_K_M 2.6 GiB · fits · files
Q5_K_M 2.9 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 4.8 NVIDIA Nemotron 3 Nano 4B · Arena unknown · Model page

LFM2.5-24B-A2B

LiquidAI/LFM2.5-24B-A2B

2026-10-1324B · 2B active

AA index—unranked

Arena—

Best file & memory

Q8_023.7 GiB fileFits · conditional

24.9 of 96.0 GiB+8K ctx · max ≈128K

Prompt~4,783 t/sEstimated

Generate~74.5 t/sEstimated

First token · 4K prompt~856 msEstimated

Memory budget

Weights file	23.74 GiB	LiquidAI/LFM2.5-24B-A2B-GGUF HF files
KV cache · 8K	0.16 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	24.90 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈128K tokens (native 128K).

Other GGUF files

Q4_K_M 13.5 GiB · fits · files
Q5_K_M 15.9 GiB · fits · files
FP16 44.7 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA index unknown · Arena unknown · Model page

MiniCPM-V 4.7 35B-A3B

openbmb/MiniCPM-V-4.7-35B-A3B

2026-10-1235B · 3B active

AA index—unranked

Arena—

Best file & memory

Q8_034.4 GiB fileFits · conditional

35.5 of 96.0 GiB+8K ctx · max ≈256K

Prompt~3,188 t/sEstimated

Generate~50 t/sEstimated

First token · 4K prompt~1.3 sEstimated

Memory budget

Weights file	34.37 GiB	openbmb/MiniCPM-V-4.7-35B-A3B-gguf HF files
KV cache · 8K	0.16 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	35.52 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈256K tokens (native 256K).

Other GGUF files

Q4_K_M 19.7 GiB · fits · files
Q5_K_M 23.0 GiB · fits · files
FP16 64.6 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA index unknown · Arena unknown · Model page

Mellum2.1-12B-A2.5B-Thinking

JetBrains/Mellum2.1-12B-A2.5B-Thinking

2026-10-0812B · 2.5B active

AA index—unranked

Arena—

Best file & memory

Q8_012.0 GiB fileFits

13.2 of 96.0 GiB8K ctx · max ≈128K

Prompt~3,826 t/sEstimated

Generate~58.7 t/sEstimated

First token · 4K prompt~1.1 sEstimated

Memory budget

Weights file	12.04 GiB	JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF HF files
KV cache · 8K	0.15 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	13.19 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈128K tokens (native 128K).

Other GGUF files

Q4_K_M 7.5 GiB · fits · files
Q5_K_M 9.3 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA index unknown · Arena unknown · Model page

Aleph Alpha Kolibri-1

Aleph-Alpha/Kolibri-1

2026-10-0378B · 3.5B active

AA index—unranked

Arena—

Best file & memory

Q8_077.4 GiB fileFits · conditional

78.6 of 96.0 GiB+8K ctx · max ≈256K

Prompt~2,764 t/sEstimated

Generate~42.9 t/sEstimated

First token · 4K prompt~1.5 sEstimated

Memory budget

Weights file	77.43 GiB	Hob-forge/Kolibri-1-GGUF · 2 files HF files
KV cache · 8K	0.20 GiB	BF16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	78.62 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈256K tokens (native 256K).

Other GGUF files

Q4_K_M 44.2 GiB · fits · files
Q5_K_M 51.8 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA index unknown · Arena unknown · Model page

MiMo-V2.6-Distill-Qwen-9B

XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B

2026-09-229B dense

AA index—unranked

Arena—

Best file & memory

Q8_08.9 GiB fileFits · conditional

10.1 of 96.0 GiB+8K ctx · max ≈256K

Prompt~2,478 t/sEstimated

Generate~23.1 t/sEstimated

First token · 4K prompt~1.7 sEstimated

Memory budget

Weights file	8.89 GiB	bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF HF files
KV cache · 8K	0.25 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	10.14 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈256K tokens (native 256K).

Other GGUF files

Q4_K_M 5.4 GiB · fits · files
Q5_K_M 6.4 GiB · fits · files
FP16 16.7 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA index unknown · Arena unknown · Model page

K2-Horizon-0.9B

IFM/K2-Horizon-0.9B

2026-09-03size unknown

AA index—unranked

Arena—

Best file & memory

Q8_01.1 GiB fileFits

2.5 of 96.0 GiB8K ctx · max ≈128K

Prompt—Unknown

Generate~192 t/sEstimated

First token · 4K prompt—Unknown

Memory budget

Weights file	1.07 GiB	IFM/K2-Horizon-0.9B-GGUF HF files
KV cache · 8K	0.44 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	2.51 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈128K tokens (native 128K).

Other GGUF files

Q4_K_M 0.6 GiB · fits · files
Q5_K_M 0.7 GiB · fits · files
FP16 2.0 GiB · fits · files

Where the speeds come from

Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token unknown: needs a prefill value.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA index unknown · Arena unknown · Model page

Ornith-1.5-9B

ornith-ai/Ornith-1.5-9B

2026-08 (day unknown)9B dense

AA index—unranked

Arena—

Best file & memory

Q8_09.1 GiB fileFits · conditional

10.4 of 96.0 GiB+8K ctx · max ≈256K

Prompt~2,478 t/sEstimated

Generate~22.5 t/sEstimated

First token · 4K prompt~1.7 sEstimated

Memory budget

Weights file	9.11 GiB	ornith-ai/Ornith-1.5-9B-GGUF HF files
KV cache · 8K	0.25 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	10.36 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈256K tokens (native 256K).

Other GGUF files

Q4_K_M 5.4 GiB · fits · files
Q5_K_M 6.2 GiB · fits · files
FP16 16.7 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA index unknown · Arena unknown · Model page

Poolside Laguna S 2.1

poolside/Laguna-S-2.1

2026-07-21118B · 8B active

AA index—unranked

Arena—

Best file & memory

Q4_K_M89.4 GiB fileFits · conditional

90.9 of 96.0 GiB+8K ctx · max ≈107K

Prompt~1,196 t/sEstimated

Generate12.6 t/sMeasured

First token · 4K prompt~3.4 sEstimated

Memory budget

Weights file	89.44 GiB	poolside/Laguna-S-2.1-GGUF HF files
KV cache · 8K	0.45 GiB	BF16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	90.88 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈107K tokens (native 1M).

Other GGUF files

Q8_0 120 GiB · too big · files
FP16 219 GiB · too big · files
BF16 219 GiB · too big · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate measured 12.6 t/s: Ollama (llama.cpp CUDA) · quant label Q4_K_M · decode tokens/s as stated on the model card (prompt/length not stated) · 2026-07-21 · huggingface.co. Reported by the source; build and settings may differ from yours.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA index unknown · Arena unknown · Model page

Qwen3.6-27B

Qwen/Qwen3.6-27B

2026-04-2227B dense

AA index—unranked

Arena—

Best file & memory

Q8_026.6 GiB fileFits · conditional

28.1 of 96.0 GiB+8K ctx · max ≈256K

Prompt~826 t/sEstimated

Generate~7.7 t/sEstimated

First token · 4K prompt~5.0 sEstimated

Memory budget

Weights file	26.63 GiB	unsloth/Qwen3.6-27B-GGUF HF files
KV cache · 8K	0.50 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	28.13 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈256K tokens (native 256K).

Other GGUF files

Q4_K_M 15.7 GiB · fits · files
Q5_K_M 18.2 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA index unknown · Arena unknown · Model page

Qwen3.5-0.8B

Qwen/Qwen3.5-0.8B

2026-03-020.8B dense

AA index—unranked

Arena—

Best file & memory

Q8_00.8 GiB fileFits · conditional

1.8 of 96.0 GiB+8K ctx · max ≈256K

Prompt~27,881 t/sEstimated

Generate~271 t/sEstimated

First token · 4K prompt~147 msEstimated

Memory budget

Weights file	0.76 GiB	unsloth/Qwen3.5-0.8B-GGUF HF files
KV cache · 8K	0.09 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	1.85 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈256K tokens (native 256K).

Other GGUF files

Q4_K_M 0.5 GiB · fits · files
Q5_K_M 0.5 GiB · fits · files
FP16 1.4 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA index unknown · Arena unknown · Model page

Qwen3.5-2B

Qwen/Qwen3.5-2B

2026-03-022B dense

AA index—unranked

Arena—

Best file & memory

Q8_01.9 GiB fileFits · conditional

3.0 of 96.0 GiB+8K ctx · max ≈256K

Prompt~11,153 t/sEstimated

Generate~110 t/sEstimated

First token · 4K prompt~367 msEstimated

Memory budget

Weights file	1.87 GiB	unsloth/Qwen3.5-2B-GGUF HF files
KV cache · 8K	0.09 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	2.97 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈256K tokens (native 256K).

Other GGUF files

Q4_K_M 1.2 GiB · fits · files
Q5_K_M 1.3 GiB · fits · files
FP16 3.5 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA index unknown · Arena unknown · Model page

Qwen3.5-27B

Qwen/Qwen3.5-27B

2026-02-2427B dense

AA index—unranked

Arena—

Best file & memory

Q8_026.6 GiB fileFits · conditional

28.1 of 96.0 GiB+8K ctx · max ≈256K

Prompt~826 t/sEstimated

Generate~7.7 t/sEstimated

First token · 4K prompt~5.0 sEstimated

Memory budget

Weights file	26.63 GiB	unsloth/Qwen3.5-27B-GGUF HF files
KV cache · 8K	0.50 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	28.13 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈256K tokens (native 256K).

Other GGUF files

Q4_K_M 15.6 GiB · fits · files
Q5_K_M 18.3 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA index unknown · Arena unknown · Model page

Qwen3.5-35B-A3B

Qwen/Qwen3.5-35B-A3B

2026-02-2435B · 3B active

AA index—unranked

Arena—

Best file & memory

Q8_034.4 GiB fileFits · conditional

35.5 of 96.0 GiB+8K ctx · max ≈256K

Prompt~3,188 t/sEstimated

Generate~50 t/sEstimated

First token · 4K prompt~1.3 sEstimated

Memory budget

Weights file	34.37 GiB	unsloth/Qwen3.5-35B-A3B-GGUF HF files
KV cache · 8K	0.16 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	35.52 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈256K tokens (native 256K).

Other GGUF files

Q4_K_M 20.5 GiB · fits · files
Q5_K_M 24.4 GiB · fits · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA index unknown · Arena unknown · Model page

May fit: KV-cache size unknown 16

Weights and runtime fit, but the cache for this attention type is not established. Ranked by AA index.

K2-Horizon-MoVA-36B-A4B

IFM/K2-Horizon-MoVA-36B-A4B

2026-09-0336B · 4B active

AA index25.3rank #22

Arena—

Best file & memory

Q8_037.1 GiB fileCache unknown

38.1 of 96.0 GiB+8K ctx

Prompt~2,391 t/sEstimated

Generate~35.7 t/sEstimated

First token · 4K prompt~1.7 sEstimated

Memory budget

Weights file	37.10 GiB	IFM/K2-Horizon-MoVA-36B-A4B-GGUF HF files
KV cache · 8K	—	Unknown
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	38.10 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: unknown (native 512K).

Other GGUF files

Q4_K_M 20.8 GiB · cache unknown · files
Q5_K_M 24.6 GiB · cache unknown · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

KV-cache size could not be established from the config (compressed or backend-specific attention).
Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 25.3 K2 Horizon MoVA 36B A4B · Arena unknown · Model page

Ling-3.0-flash-VL

inclusionAI/Ling-3.0-flash-VL

2026-09-09124B · 5.5B active

AA index24.6rank #26

Arena—

Best file & memory

Q5_K_M89.3 GiB fileCache unknown

90.3 of 96.0 GiB+8K ctx

Prompt~1,739 t/sEstimated

Generate~37.2 t/sEstimated

First token · 4K prompt~2.4 sEstimated

Memory budget

Weights file	89.30 GiB	bartowski/Ling-3.0-flash-VL-GGUF · 3 files HF files
KV cache · 8K	—	Unknown
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	90.30 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: unknown (native 128K).

Other GGUF files

Q4_K_M 73.3 GiB · cache unknown · files
Q8_0 123 GiB · too big · files
BF16 232 GiB · too big · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

KV-cache size could not be established from the config (compressed or backend-specific attention).
Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 24.6 Ling-3.0-flash-VL · Arena unknown · Model page

Ling-3.0-flash-Fin

inclusionAI/Ling-3.0-flash-Fin

2026 (month unknown)124B · 5.1B active

AA index22.6rank #28

Arena—

Best file & memory

Q5_K_M84.4 GiB fileCache unknown

85.4 of 96.0 GiB+8K ctx

Prompt~1,875 t/sEstimated

Generate~42.4 t/sEstimated

First token · 4K prompt~2.2 sEstimated

Memory budget

Weights file	84.43 GiB	bartowski/Ling-3.0-flash-Fin-GGUF · 3 files HF files
KV cache · 8K	—	Unknown
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	85.43 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: unknown (native 256K).

Other GGUF files

Q4_K_M 72.5 GiB · cache unknown · files
Q8_0 126 GiB · too big · files
BF16 238 GiB · too big · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

KV-cache size could not be established from the config (compressed or backend-specific attention).
Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 22.6 Ling-3.0-flash-Fin · Arena unknown · Model page

Gemma 4 26B A4B (it)

google/gemma-4-26B-A4B-it

2026-04-0225B · 3.8B active

AA index16.7 AA estimaterank #39

Arena1,434

Best file & memory

Q8_025.0 GiB fileCache unknown

26.0 of 96.0 GiB+8K ctx

Prompt~2,517 t/sEstimated

Generate~39.1 t/sEstimated

First token · 4K prompt~1.6 sEstimated

Memory budget

Weights file	25.02 GiB	unsloth/gemma-4-26B-A4B-it-GGUF HF files
KV cache · 8K	—	Unknown
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	26.02 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: unknown (native 256K).

Other GGUF files

Q4_K_M 15.6 GiB · cache unknown · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

KV-cache size could not be established from the config (compressed or backend-specific attention).
Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 16.7 AA estimate Gemma 4 26B A4B (Reasoning) · Arena 1,434 · Model page

Gemma 4 31B (it)

google/gemma-4-31B-it

2026-04-0231B dense

AA index14.7rank #42

Arena1,443

Best file & memory

Q8_030.4 GiB fileCache unknown

31.4 of 96.0 GiB+8K ctx

Prompt~727 t/sEstimated

Generate~6.8 t/sEstimated

First token · 4K prompt~5.6 sEstimated

Memory budget

Weights file	30.39 GiB	unsloth/gemma-4-31B-it-GGUF HF files
KV cache · 8K	—	Unknown
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	31.39 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: unknown (native 256K).

Other GGUF files

Q4_K_M 17.1 GiB · cache unknown · files
Q5_K_M 20.2 GiB · cache unknown · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

KV-cache size could not be established from the config (compressed or backend-specific attention).
Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 14.7 Gemma 4 31B (Reasoning) · Arena 1,443 · Model page

Gemma 4 12B (it)

google/gemma-4-12B-it

2026-06-0312B dense

AA index14.2 AA estimaterank #44

Arena—

Best file & memory

Q8_011.8 GiB fileCache unknown

12.8 of 96.0 GiB+8K ctx

Prompt~1,867 t/sEstimated

Generate~17.4 t/sEstimated

First token · 4K prompt~2.2 sEstimated

Memory budget

Weights file	11.80 GiB	bartowski/gemma-4-12B-it-GGUF HF files
KV cache · 8K	—	Unknown
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	12.80 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: unknown (native 256K).

Other GGUF files

Q4_K_M 7.1 GiB · cache unknown · files
Q5_K_M 8.2 GiB · cache unknown · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

KV-cache size could not be established from the config (compressed or backend-specific attention).
Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 14.2 AA estimate Gemma 4 12B (Reasoning) · Arena unknown · Model page

LongCat-Flash-Lite

meituan-longcat/LongCat-Flash-Lite

2026-01 (day unknown)69B · 3B active

AA index11.5 AA estimaterank #54

Arena—

Best file & memory

Q8_067.9 GiB fileCache unknown

68.9 of 96.0 GiB+8K ctx

Prompt~3,188 t/sEstimated

Generate~49.6 t/sEstimated

First token · 4K prompt~1.3 sEstimated

Memory budget

Weights file	67.87 GiB	InquiringMinds-AI/LongCat-Flash-Lite-GGUF HF files
KV cache · 8K	—	Unknown
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	68.87 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: unknown (native 320K).

Other GGUF files

Q4_K_M 37.5 GiB · cache unknown · files
Q5_K_M 44.7 GiB · cache unknown · files
BF16 128 GiB · too big · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

KV-cache size could not be established from the config (compressed or backend-specific attention).
Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 11.5 AA estimate LongCat Flash Lite · Arena unknown · Model page

Mistral Small 4 (119B-2603)

mistralai/Mistral-Small-4-119B-2603

2026-03-16119B · 6B active

AA index11.3rank #55

Arena—

Best file & memory

Q5_K_M79.2 GiB fileCache unknown

80.2 of 96.0 GiB+8K ctx

Prompt~1,594 t/sEstimated

Generate~36.9 t/sEstimated

First token · 4K prompt~2.6 sEstimated

Memory budget

Weights file	79.20 GiB	bartowski/mistralai_Mistral-Small-4-119B-2603-GGUF · 3 files HF files
KV cache · 8K	—	Unknown
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	80.20 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: unknown (native 1M).

Other GGUF files

Q4_K_M 67.6 GiB · cache unknown · files
Q8_0 118 GiB · too big · files
BF16 222 GiB · too big · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

KV-cache size could not be established from the config (compressed or backend-specific attention).
Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 11.3 Mistral Small 4 (Reasoning) · Arena unknown · Model page

Ling-3.0-tiny

inclusionAI/Ling-3.0-tiny

2026-08 (day unknown)7.9B · 1.3B active

AA index11.1rank #57

Arena—

Best file & memory

Q8_07.8 GiB fileCache unknown

8.8 of 96.0 GiB+8K ctx

Prompt~7,358 t/sEstimated

Generate~114 t/sEstimated

First token · 4K prompt~557 msEstimated

Memory budget

Weights file	7.83 GiB	inclusionAI/Ling-3.0-tiny-GGUF HF files
KV cache · 8K	—	Unknown
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	8.83 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: unknown (native 128K).

Other GGUF files

Q4_K_M 4.5 GiB · cache unknown · files
Q5_K_M 5.2 GiB · cache unknown · files
FP16 14.7 GiB · cache unknown · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

KV-cache size could not be established from the config (compressed or backend-specific attention).
Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 11.1 Ling 3.0 Tiny · Arena unknown · Model page

Nemotron 3 Nano Omni 30B A3B Reasoning

nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16

2026-04-2830B · 3B active

AA index10.3 AA estimaterank #60

Arena—

Best file & memory

Q8_031.3 GiB fileCache unknown

32.3 of 96.0 GiB+8K ctx

Prompt~3,188 t/sEstimated

Generate~47.1 t/sEstimated

First token · 4K prompt~1.3 sEstimated

Memory budget

Weights file	31.28 GiB	unsloth/NVIDIA-Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF HF files
KV cache · 8K	—	Unknown
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	32.28 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: unknown (native 256K).

Other GGUF files

Q4_K_M 22.8 GiB · cache unknown · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

KV-cache size could not be established from the config (compressed or backend-specific attention).
Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 10.3 AA estimate Nemotron 3 Nano Omni 30B A3B Reasoning · Arena unknown · Model page

DiffusionGemma 26B A4B

google/diffusiongemma-26B-A4B-it

2026-06-1025B · 3.8B active

AA index9.5 AA estimaterank #62

Arena—

Best file & memory

Q8_025.0 GiB fileCache unknown

26.0 of 96.0 GiB+8K ctx

Prompt~2,517 t/sEstimated

Generate~39 t/sEstimated

First token · 4K prompt~1.6 sEstimated

Memory budget

Weights file	25.03 GiB	unsloth/diffusiongemma-26B-A4B-it-GGUF HF files
KV cache · 8K	—	Unknown
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	26.03 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: unknown (native 256K).

Other GGUF files

Q4_K_M 15.7 GiB · cache unknown · files
Q5_K_M 17.8 GiB · cache unknown · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

KV-cache size could not be established from the config (compressed or backend-specific attention).
Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 9.5 AA estimate DiffusionGemma 26B A4B · Arena unknown · Model page

Gemma 4 E4B (it)

google/gemma-4-E4B-it

2026-04-028B dense

AA index8.9rank #66

Arena—

Best file & memory

Q8_07.6 GiB fileCache unknown

8.6 of 96.0 GiB+8K ctx

Prompt~2,788 t/sEstimated

Generate~26.9 t/sEstimated

First token · 4K prompt~1.5 sEstimated

Memory budget

Weights file	7.63 GiB	unsloth/gemma-4-E4B-it-GGUF HF files
KV cache · 8K	—	Unknown
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	8.63 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: unknown (native 128K).

Other GGUF files

Q4_K_M 4.6 GiB · cache unknown · files
Q5_K_M 5.1 GiB · cache unknown · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

KV-cache size could not be established from the config (compressed or backend-specific attention).
Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 8.9 Gemma 4 E4B (Reasoning) · Arena unknown · Model page

Sarvam 105B

sarvamai/sarvam-105b

2026-03-06105B · 10B active

AA index8.8 AA estimaterank #68

Arena—

Best file & memory

Q4_K_M59.8 GiB fileCache unknown

60.8 of 96.0 GiB+8K ctx

Prompt~929 t/sEstimated

Generate~25.1 t/sEstimated

First token · 4K prompt~4.4 sEstimated

Memory budget

Weights file	59.80 GiB	limegreenpeper1/sarvam-105B-GGUF · 14 files HF files
KV cache · 8K	—	Unknown
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	60.80 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: unknown (native 128K).

Other GGUF files

Q8_0 105 GiB · too big · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

KV-cache size could not be established from the config (compressed or backend-specific attention).
Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 8.8 AA estimate Sarvam 105B (High) · Arena unknown · Model page

Gemma 4 E2B

google/gemma-4-E2B-it

2026-04-025.1B dense

AA index7.8 AA estimaterank #72

Arena—

Best file & memory

Q8_04.7 GiB fileCache unknown

5.7 of 96.0 GiB+8K ctx

Prompt~4,374 t/sEstimated

Generate~43.7 t/sEstimated

First token · 4K prompt~937 msEstimated

Memory budget

Weights file	4.70 GiB	unsloth/gemma-4-E2B-it-GGUF HF files
KV cache · 8K	—	Unknown
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	5.70 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: unknown (native 128K).

Other GGUF files

Q4_K_M 2.9 GiB · cache unknown · files
Q5_K_M 3.1 GiB · cache unknown · files

Where the speeds come from

Prompt estimated from median of 9 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 7 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

KV-cache size could not be established from the config (compressed or backend-specific attention).
Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 7.8 AA estimate Gemma 4 E2B (Reasoning) · Arena unknown · Model page

Ling-3.0-flash

inclusionAI/Ling-3.0-flash

2026-08-07124B · 5.1B active

AA index—unranked

Arena—

Best file & memory

Q5_K_M84.4 GiB fileCache unknown

85.4 of 96.0 GiB+8K ctx

Prompt~1,875 t/sEstimated

Generate~42.4 t/sEstimated

First token · 4K prompt~2.2 sEstimated

Memory budget

Weights file	84.43 GiB	bartowski/Ling-3.0-flash-GGUF · 3 files HF files
KV cache · 8K	—	Unknown
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	85.43 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: unknown (native 256K).

Other GGUF files

Q4_K_M 72.5 GiB · cache unknown · files
Q8_0 126 GiB · too big · files
BF16 238 GiB · too big · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

KV-cache size could not be established from the config (compressed or backend-specific attention).
Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA index unknown · Arena unknown · Model page

GLM-4.7-Flash

zai-org/GLM-4.7-Flash

2026-01 (day unknown)30B · 3B active

AA index—unranked

Arena—

Best file & memory

Q8_029.7 GiB fileCache unknown

30.7 of 96.0 GiB+8K ctx

Prompt~3,188 t/sEstimated

Generate~49.7 t/sEstimated

First token · 4K prompt~1.3 sEstimated

Memory budget

Weights file	29.66 GiB	unsloth/GLM-4.7-Flash-GGUF HF files
KV cache · 8K	—	Unknown
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	30.66 GiB	of 96.0 GiB usable (75% of 128 GB) · 0.5 GiB spare margin required

Max context in free memory: unknown (native 198K).

Other GGUF files

Q4_K_M 17.1 GiB · cache unknown · files
Q5_K_M 19.9 GiB · cache unknown · files

Where the speeds come from

Prompt estimated from median of 14 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
Generate estimated from median of 13 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

Caveats

KV-cache size could not be established from the config (compressed or backend-specific attention).
Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA index unknown · Arena unknown · Model page

26 models too big for DGX Spark (128 GB)

GLM-5.3Q8_0 needs 748 GiB+ (KV unknown)
GLM-5.3-FlashQ8_0 needs 319 GiB+ (KV unknown)
Ling 3.1 FlashQ4_K_M needs 335 GiB+ (KV unknown)
Qwen3.8-2.4T-A95BQ8_0 needs 2,424 GiB
Qwen3.8-Flash-NextQ8_0 needs 177 GiB+ (KV unknown)
Muse Spark Open (816B-A42B)Q8_0 needs 813 GiB+ (KV unknown)
Motif 3Q4_K_M needs 185 GiB+ (KV unknown)
K2-Horizon-375B-A23BQ4_K_M needs 225 GiB
Reflection Beam 1.0 501B-A23BQ8_0 needs 498 GiB
MiniMax-M3Q4_K_M needs 245 GiB+ (KV unknown)
Nex-N2-Pro (Qwen3.5-397B-A17B based)Q4_K_M needs 227 GiB
NVIDIA Nemotron 3.5 Super VL 120B-A12BQ8_0 needs 124 GiB
Kimi K2.7 CodeQ4_K_M needs 579 GiB+ (KV unknown)
Inkling-SmallQ8_0 needs 263 GiB
Hy3Q4_K_M needs 174 GiB
InklingQ8_0 needs 800 GiB
Solar Open 2 (250B-A15B)Q8_0 needs 250 GiB
NVIDIA Nemotron 3 Ultra 550B-A55BQ4_K_M needs 323 GiB
Qwen3.5-397B-A17BQ4_K_M needs 229 GiB
Command A+ (05-2026)Q4_K_M needs 125 GiB
Trinity-Large-ThinkingQ4_K_M needs 228 GiB
Hy4-previewQ4_K_M needs 437 GiB+ (KV unknown)
GLM-5.2Q8_0 needs 748 GiB+ (KV unknown)
Step 3.7 FlashQ8_0 needs 197 GiB+ (KV unknown)
NVIDIA Nemotron 3 Super 120B-A12BQ8_0 needs 121 GiB
MiniMax-M2.7Q4_K_M needs 133 GiB

Specs & sources

Memory	128 GB LPDDR5X	source
Bandwidth	273 GB/s	source
FP16	unknown	NVIDIA publishes only 'Up to 1 PFLOP FP4' for GB10; no FP16 (dense or sparse) figure on the product page.
Power	140 W	source

Measured speeds

122 reported observations

On this device. Reported by the linked sources; not reproduced by LocalFit.

Open in speed database Submit yours

Doesn't fit? Rent a GPU

RunpodPlaceholderCloud GPU pods you can rent for running larger models.
Vast.aiPlaceholderGPU rental marketplace with many hosts and GPU types.
LambdaPlaceholderCloud GPU instances for AI training and inference.
TensorDockPlaceholderGPU cloud marketplace with consumer and datacenter GPUs.

Sponsored placeholder links: LocalFit may earn a commission if these become affiliate links. They never influence fit results, estimates or picks.