NVIDIA GeForce RTX 4060: which models fit?

RTX 4090 RTX 5090 RTX 3090 RTX 4060 Ti (16 GB)RTX 5070 Ti Radeon RX 7900 XTX M4 Max 128 GB M3 Ultra 512 GB DGX Spark (128 GB)All devices

NVIDIA · GPU

NVIDIA GeForce RTX 4060

Memory8 GB GDDR6BandwidthUnknown not published in reviewed sourcesPlanned for models8.0 GiB all VRAM usableFilesGGUF llama.cpp format

Editorial pick · 8 GB GPUsK2-Horizon-7B Q4_K_M All picks

Budget = file + KV cache + 1 GiB runtime + 0.5 GiB spare. Macs: 75% of memory. Details

20 models fit RTX 4060 · 8K context

Ranked by Artificial Analysis Intelligence Index, unranked models last. Arena is shown separately. How ranking works

Intelligence

Artificial Analysis Intelligence Index · top 10 models that fit RTX 4060 · higher is better

AA estimate

Generation speed

Tokens per second on RTX 4060, best file for each of the same models · higher is better

No generation value for these models on RTX 4060: no measured report, and bandwidth, active size or backend calibration is missing.

WeightsKV cacheRuntimeFreeMeasuredlinks to its sourceEstimatedshown with ~Unknownnever zero

K2-Horizon-7B

IFM/K2-Horizon-7B

2026-09-03size unknown

AA index20.6rank #33

Arena—

Best file & memory

Q4_K_M5.2 GiB fileFits

7.3 of 8.0 GiB8K ctx · max ≈9K

Prompt—Unknown

Generate—Unknown

First token · 4K prompt—Unknown

Memory budget

Weights file	5.21 GiB	IFM/K2-Horizon-7B-GGUF HF files
KV cache · 8K	1.13 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	7.33 GiB	of 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈9K tokens (native 512K).

Other GGUF files

Q5_K_M 6.0 GiB · tight · files
Q8_0 8.9 GiB · too big · files
FP16 16.8 GiB · too big · files

Where the speeds come from

Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
Generate unknown: no measured report, and bandwidth, active size or backend calibration is missing.
First token unknown: needs a prefill value.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 20.6 K2 Horizon 7B · Arena unknown · Model page

K2-Horizon-3.7B

IFM/K2-Horizon-3.7B

2026-09-03size unknown

AA index15.6rank #41

Arena—

Best file & memory

Q8_05.0 GiB fileFits

7.1 of 8.0 GiB8K ctx · max ≈11K

Prompt—Unknown

Generate—Unknown

First token · 4K prompt—Unknown

Memory budget

Weights file	5.02 GiB	IFM/K2-Horizon-3.7B-GGUF HF files
KV cache · 8K	1.13 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	7.14 GiB	of 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈11K tokens (native 512K).

Other GGUF files

Q4_K_M 2.9 GiB · fits · files
Q5_K_M 3.4 GiB · fits · files
FP16 9.4 GiB · too big · files

Where the speeds come from

Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
Generate unknown: no measured report, and bandwidth, active size or backend calibration is missing.
First token unknown: needs a prefill value.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 15.6 K2 Horizon 3.7B · Arena unknown · Model page

Qwen3.5-9B

Qwen/Qwen3.5-9B

2026-03-029B dense

AA index13.3 AA estimaterank #45

Arena—

Best file & memory

Q5_K_M6.1 GiB fileFits · conditional

7.4 of 8.0 GiB+8K ctx · max ≈12K

Prompt—Unknown

Generate—Unknown

First token · 4K prompt—Unknown

Memory budget

Weights file	6.13 GiB	unsloth/Qwen3.5-9B-GGUF HF files
KV cache · 8K	0.25 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	7.38 GiB	of 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈12K tokens (native 256K).

Other GGUF files

Q4_K_M 5.3 GiB · fits · files
Q8_0 8.9 GiB · too big · files

Where the speeds come from

Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
Generate unknown: no measured report, and bandwidth, active size or backend calibration is missing.
First token unknown: needs a prefill value.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 13.3 AA estimate Qwen3.5 9B (Non-reasoning) · Arena unknown · Model page

Qwen3.5-4B

Qwen/Qwen3.5-4B

2026-03-024B dense

AA index13.1 AA estimaterank #48

Arena—

Best file & memory

Q8_04.2 GiB fileFits · conditional

5.4 of 8.0 GiB+8K ctx · max ≈74K

Prompt—Unknown

Generate—Unknown

First token · 4K prompt—Unknown

Memory budget

Weights file	4.17 GiB	unsloth/Qwen3.5-4B-GGUF HF files
KV cache · 8K	0.25 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	5.42 GiB	of 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈74K tokens (native 256K).

Other GGUF files

Q4_K_M 2.6 GiB · fits · files
Q5_K_M 2.9 GiB · fits · files

Where the speeds come from

Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
Generate unknown: no measured report, and bandwidth, active size or backend calibration is missing.
First token unknown: needs a prefill value.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 13.1 AA estimate Qwen3.5 4B (Reasoning) · Arena unknown · Model page

MiniCPM5-2B

openbmb/MiniCPM5-2B

2026-09-072.5B dense

AA index12.5rank #51

Arena—

Best file & memory

Q8_02.5 GiB fileFits

3.8 of 8.0 GiB8K ctx · max ≈98K

Prompt—Unknown

Generate—Unknown

First token · 4K prompt—Unknown

Memory budget

Weights file	2.50 GiB	openbmb/MiniCPM5-2B-GGUF HF files
KV cache · 8K	0.33 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	3.82 GiB	of 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈98K tokens (native 128K).

Other GGUF files

Q4_K_M 1.5 GiB · fits · files
Q5_K_M 1.8 GiB · fits · files
FP16 4.7 GiB · fits · files

Where the speeds come from

Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
Generate unknown: no measured report, and bandwidth, active size or backend calibration is missing.
First token unknown: needs a prefill value.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 12.5 MiniCPM5-2B · Arena unknown · Model page

Granite 4.2 8B

ibm-granite/granite-4.2-8b

2026-08-258B dense

AA index11.1rank #56

Arena1,320

Best file & memory

Q4_K_M5.0 GiB fileFits

7.2 of 8.0 GiB8K ctx · max ≈10K

Prompt—Unknown

Generate—Unknown

First token · 4K prompt—Unknown

Memory budget

Weights file	4.98 GiB	ibm-granite/granite-4.2-8b-GGUF HF files
KV cache · 8K	1.25 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	7.23 GiB	of 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈10K tokens (native 128K).

Other GGUF files

Q5_K_M 5.8 GiB · tight · files
Q8_0 8.7 GiB · too big · files
FP16 16.4 GiB · too big · files

Where the speeds come from

Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
Generate unknown: no measured report, and bandwidth, active size or backend calibration is missing.
First token unknown: needs a prefill value.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 11.1 Granite 4.2 8B · Arena 1,320 · Model page

G9v3-3B

AI9Stars/G9v3-3B

2026-07 (day unknown)3B dense

AA index10.8 AA estimaterank #58

Arena—

Best file & memory

Q8_03.0 GiB fileFits

4.4 of 8.0 GiB8K ctx · max ≈70K

Prompt—Unknown

Generate—Unknown

First token · 4K prompt—Unknown

Memory budget

Weights file	2.96 GiB	mradermacher/G9v3-3B-GGUF HF files
KV cache · 8K	0.41 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	4.37 GiB	of 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈70K tokens (native 128K).

Other GGUF files

Q4_K_M 1.7 GiB · fits · files
Q5_K_M 2.0 GiB · fits · files
FP16 5.6 GiB · fits · files

Where the speeds come from

Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
Generate unknown: no measured report, and bandwidth, active size or backend calibration is missing.
First token unknown: needs a prefill value.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 10.8 AA estimate G9v3-3B · Arena unknown · Model page

Granite 4.2 3B

ibm-granite/granite-4.2-3b

2026-08-253B dense

AA index9.1rank #64

Arena1,293

Best file & memory

Q8_03.6 GiB fileFits

5.3 of 8.0 GiB8K ctx · max ≈37K

Prompt—Unknown

Generate—Unknown

First token · 4K prompt—Unknown

Memory budget

Weights file	3.63 GiB	ibm-granite/granite-4.2-3b-GGUF HF files
KV cache · 8K	0.63 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	5.25 GiB	of 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈37K tokens (native 128K).

Other GGUF files

Q4_K_M 2.1 GiB · fits · files
Q5_K_M 2.4 GiB · fits · files
FP16 6.8 GiB · too big · files

Where the speeds come from

Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
Generate unknown: no measured report, and bandwidth, active size or backend calibration is missing.
First token unknown: needs a prefill value.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 9.1 Granite 4.2 3B · Arena 1,293 · Model page

MiniCPM5-1B

openbmb/MiniCPM5-1B

2026-05-191.1B dense

AA index8.8 AA estimaterank #67

Arena—

Best file & memory

Q8_01.1 GiB fileFits

2.3 of 8.0 GiB8K ctx · max ≈128K

Prompt—Unknown

Generate—Unknown

First token · 4K prompt—Unknown

Memory budget

Weights file	1.07 GiB	openbmb/MiniCPM5-1B-GGUF HF files
KV cache · 8K	0.19 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	2.26 GiB	of 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈128K tokens (native 128K).

Other GGUF files

Q4_K_M 0.6 GiB · fits · files
Q5_K_M 0.7 GiB · fits · files
FP16 2.0 GiB · fits · files

Where the speeds come from

Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
Generate unknown: no measured report, and bandwidth, active size or backend calibration is missing.
First token unknown: needs a prefill value.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 8.8 AA estimate MiniCPM5-1B (Reasoning) · Arena unknown · Model page

Nanbeige4.1-3B

Nanbeige/Nanbeige4.1-3B

2026-02 (day unknown)3B dense

AA index8.4 AA estimaterank #69

Arena—

Best file & memory

Q8_03.9 GiB fileFits

5.4 of 8.0 GiB8K ctx · max ≈42K

Prompt—Unknown

Generate—Unknown

First token · 4K prompt—Unknown

Memory budget

Weights file	3.90 GiB	mradermacher/Nanbeige4.1-3B-GGUF HF files
KV cache · 8K	0.50 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	5.40 GiB	of 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈42K tokens (native 256K).

Other GGUF files

Q4_K_M 2.3 GiB · fits · files
Q5_K_M 2.6 GiB · fits · files
FP16 7.3 GiB · too big · files

Where the speeds come from

Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
Generate unknown: no measured report, and bandwidth, active size or backend calibration is missing.
First token unknown: needs a prefill value.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 8.4 AA estimate Nanbeige4.1-3B · Arena unknown · Model page

LFM2.5-2.6B

LiquidAI/LFM2.5-2.6B

2026-08-042.7B dense

AA index8.4 AA estimaterank #70

Arena—

Best file & memory

Q8_02.7 GiB fileFits · conditional

3.8 of 8.0 GiB+8K ctx · max ≈128K

Prompt—Unknown

Generate—Unknown

First token · 4K prompt—Unknown

Memory budget

Weights file	2.68 GiB	LiquidAI/LFM2.5-2.6B-GGUF HF files
KV cache · 8K	0.13 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	3.80 GiB	of 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈128K tokens (native 128K).

Other GGUF files

Q4_K_M 1.6 GiB · fits · files
Q5_K_M 1.8 GiB · fits · files
FP16 5.0 GiB · fits · files

Where the speeds come from

Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
Generate unknown: no measured report, and bandwidth, active size or backend calibration is missing.
First token unknown: needs a prefill value.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 8.4 AA estimate LFM2.5-2.6B · Arena unknown · Model page

Falcon-H1R-7B

tiiuae/Falcon-H1R-7B

2026-01-057B dense

AA index7.8 AA estimaterank #71

Arena—

Best file & memory

Q5_K_M5.0 GiB fileFits · conditional

6.4 of 8.0 GiB+8K ctx · max ≈34K

Prompt—Unknown

Generate—Unknown

First token · 4K prompt—Unknown

Memory budget

Weights file	5.02 GiB	tiiuae/Falcon-H1R-7B-GGUF HF files
KV cache · 8K	0.34 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	6.36 GiB	of 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈34K tokens (native 256K).

Other GGUF files

Q4_K_M 4.3 GiB · fits · files
Q8_0 7.5 GiB · too big · files
FP16 14.1 GiB · too big · files

Where the speeds come from

Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
Generate unknown: no measured report, and bandwidth, active size or backend calibration is missing.
First token unknown: needs a prefill value.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 7.8 AA estimate Falcon-H1R-7B · Arena unknown · Model page

Step3 VL 10B

stepfun-ai/Step3-VL-10B

2026-01 (day unknown)10B dense

AA index7.7 AA estimaterank #73

Arena—

Best file & memory

Q4_K_M4.7 GiB fileFits

6.8 of 8.0 GiB8K ctx · max ≈13K

Prompt—Unknown

Generate—Unknown

First token · 4K prompt—Unknown

Memory budget

Weights file	4.68 GiB	JamePeng2023/Step3-VL-10B-GGUF HF files
KV cache · 8K	1.13 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	6.81 GiB	of 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈13K tokens (native 64K).

Other GGUF files

Q5_K_M 5.4 GiB · tight · files
Q8_0 8.1 GiB · too big · files
FP16 15.3 GiB · too big · files

Where the speeds come from

Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
Generate unknown: no measured report, and bandwidth, active size or backend calibration is missing.
First token unknown: needs a prefill value.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 7.7 AA estimate Step3 VL 10B · Arena unknown · Model page

LFM2.5-8B-A1B

LiquidAI/LFM2.5-8B-A1B

2026-05-288.3B · 1.5B active

AA index7.2 AA estimaterank #74

Arena—

Best file & memory

Q5_K_M5.6 GiB fileFits · conditional

6.7 of 8.0 GiB+8K ctx · max ≈75K

Prompt—Unknown

Generate—Unknown

First token · 4K prompt—Unknown

Memory budget

Weights file	5.62 GiB	LiquidAI/LFM2.5-8B-A1B-GGUF HF files
KV cache · 8K	0.09 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	6.71 GiB	of 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈75K tokens (native 125K).

Other GGUF files

Q4_K_M 4.8 GiB · fits · files
Q8_0 8.4 GiB · too big · files
FP16 15.8 GiB · too big · files

Where the speeds come from

Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
Generate unknown: no measured report, and bandwidth, active size or backend calibration is missing.
First token unknown: needs a prefill value.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 7.2 AA estimate LFM2.5-8B-A1B · Arena unknown · Model page

NVIDIA Nemotron 3 Nano 4B

nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16

2026-03-164B dense

AA index4.8rank #75

Arena—

Best file & memory

Q8_03.9 GiB fileFits · conditional

5.1 of 8.0 GiB+8K ctx · max ≈164K

Prompt—Unknown

Generate—Unknown

First token · 4K prompt—Unknown

Memory budget

Weights file	3.94 GiB	unsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF HF files
KV cache · 8K	0.13 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	5.07 GiB	of 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈164K tokens (native 256K).

Other GGUF files

Q4_K_M 2.6 GiB · fits · files
Q5_K_M 2.9 GiB · fits · files

Where the speeds come from

Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
Generate unknown: no measured report, and bandwidth, active size or backend calibration is missing.
First token unknown: needs a prefill value.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 4.8 NVIDIA Nemotron 3 Nano 4B · Arena unknown · Model page

MiMo-V2.6-Distill-Qwen-9B

XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B

2026-09-229B dense

AA index—unranked

Arena—

Best file & memory

Q4_K_M5.4 GiB fileFits · conditional

6.7 of 8.0 GiB+8K ctx · max ≈34K

Prompt—Unknown

Generate—Unknown

First token · 4K prompt—Unknown

Memory budget

Weights file	5.44 GiB	bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF HF files
KV cache · 8K	0.25 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	6.69 GiB	of 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈34K tokens (native 256K).

Other GGUF files

Q5_K_M 6.4 GiB · tight · files
Q8_0 8.9 GiB · too big · files
FP16 16.7 GiB · too big · files

Where the speeds come from

Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
Generate unknown: no measured report, and bandwidth, active size or backend calibration is missing.
First token unknown: needs a prefill value.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA index unknown · Arena unknown · Model page

K2-Horizon-0.9B

IFM/K2-Horizon-0.9B

2026-09-03size unknown

AA index—unranked

Arena—

Best file & memory

Q8_01.1 GiB fileFits

2.5 of 8.0 GiB8K ctx · max ≈99K

Prompt—Unknown

Generate—Unknown

First token · 4K prompt—Unknown

Memory budget

Weights file	1.07 GiB	IFM/K2-Horizon-0.9B-GGUF HF files
KV cache · 8K	0.44 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Accounted	2.51 GiB	of 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈99K tokens (native 128K).

Other GGUF files

Q4_K_M 0.6 GiB · fits · files
Q5_K_M 0.7 GiB · fits · files
FP16 2.0 GiB · fits · files

Where the speeds come from

Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
Generate unknown: no measured report, and bandwidth, active size or backend calibration is missing.
First token unknown: needs a prefill value.

Caveats

Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA index unknown · Arena unknown · Model page

Ornith-1.5-9B

ornith-ai/Ornith-1.5-9B

2026-08 (day unknown)9B dense

AA index—unranked

Arena—

Best file & memory

Q5_K_M6.2 GiB fileFits · conditional

7.4 of 8.0 GiB+8K ctx · max ≈10K

Prompt—Unknown

Generate—Unknown

First token · 4K prompt—Unknown

Memory budget

Weights file	6.19 GiB	ornith-ai/Ornith-1.5-9B-GGUF HF files
KV cache · 8K	0.25 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	7.44 GiB	of 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈10K tokens (native 256K).

Other GGUF files

Q4_K_M 5.4 GiB · fits · files
Q8_0 9.1 GiB · too big · files
FP16 16.7 GiB · too big · files

Where the speeds come from

Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
Generate unknown: no measured report, and bandwidth, active size or backend calibration is missing.
First token unknown: needs a prefill value.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA index unknown · Arena unknown · Model page

Qwen3.5-0.8B

Qwen/Qwen3.5-0.8B

2026-03-020.8B dense

AA index—unranked

Arena—

Best file & memory

Q8_00.8 GiB fileFits · conditional

1.8 of 8.0 GiB+8K ctx · max ≈256K

Prompt—Unknown

Generate—Unknown

First token · 4K prompt—Unknown

Memory budget

Weights file	0.76 GiB	unsloth/Qwen3.5-0.8B-GGUF HF files
KV cache · 8K	0.09 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	1.85 GiB	of 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈256K tokens (native 256K).

Other GGUF files

Q4_K_M 0.5 GiB · fits · files
Q5_K_M 0.5 GiB · fits · files
FP16 1.4 GiB · fits · files

Where the speeds come from

Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
Generate unknown: no measured report, and bandwidth, active size or backend calibration is missing.
First token unknown: needs a prefill value.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA index unknown · Arena unknown · Model page

Qwen3.5-2B

Qwen/Qwen3.5-2B

2026-03-022B dense

AA index—unranked

Arena—

Best file & memory

Q8_01.9 GiB fileFits · conditional

3.0 of 8.0 GiB+8K ctx · max ≈256K

Prompt—Unknown

Generate—Unknown

First token · 4K prompt—Unknown

Memory budget

Weights file	1.87 GiB	unsloth/Qwen3.5-2B-GGUF HF files
KV cache · 8K	0.09 GiB	FP16 cache from config.json Estimated
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	2.97 GiB	of 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

Max context in free memory: ≈256K tokens (native 256K).

Other GGUF files

Q4_K_M 1.2 GiB · fits · files
Q5_K_M 1.3 GiB · fits · files
FP16 3.5 GiB · fits · files

Where the speeds come from

Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
Generate unknown: no measured report, and bandwidth, active size or backend calibration is missing.
First token unknown: needs a prefill value.

Caveats

Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA index unknown · Arena unknown · Model page

May fit: KV-cache size unknown 3

Weights and runtime fit, but the cache for this attention type is not established. Ranked by AA index.

Ling-3.0-tiny

inclusionAI/Ling-3.0-tiny

2026-08 (day unknown)7.9B · 1.3B active

AA index11.1rank #57

Arena—

Best file & memory

Q5_K_M5.2 GiB fileCache unknown

6.2 of 8.0 GiB+8K ctx

Prompt—Unknown

Generate—Unknown

First token · 4K prompt—Unknown

Memory budget

Weights file	5.25 GiB	inclusionAI/Ling-3.0-tiny-GGUF HF files
KV cache · 8K	—	Unknown
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	6.25 GiB	of 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

Max context in free memory: unknown (native 128K).

Other GGUF files

Q4_K_M 4.5 GiB · cache unknown · files
Q8_0 7.8 GiB · too big · files
FP16 14.7 GiB · too big · files

Where the speeds come from

Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
Generate unknown: no measured report, and bandwidth, active size or backend calibration is missing.
First token unknown: needs a prefill value.

Caveats

KV-cache size could not be established from the config (compressed or backend-specific attention).
Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 11.1 Ling 3.0 Tiny · Arena unknown · Model page

Gemma 4 E4B (it)

google/gemma-4-E4B-it

2026-04-028B dense

AA index8.9rank #66

Arena—

Best file & memory

Q5_K_M5.1 GiB fileCache unknown

6.1 of 8.0 GiB+8K ctx

Prompt—Unknown

Generate—Unknown

First token · 4K prompt—Unknown

Memory budget

Weights file	5.11 GiB	unsloth/gemma-4-E4B-it-GGUF HF files
KV cache · 8K	—	Unknown
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	6.11 GiB	of 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

Max context in free memory: unknown (native 128K).

Other GGUF files

Q4_K_M 4.6 GiB · cache unknown · files
Q8_0 7.6 GiB · too big · files

Where the speeds come from

Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
Generate unknown: no measured report, and bandwidth, active size or backend calibration is missing.
First token unknown: needs a prefill value.

Caveats

KV-cache size could not be established from the config (compressed or backend-specific attention).
Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 8.9 Gemma 4 E4B (Reasoning) · Arena unknown · Model page

Gemma 4 E2B

google/gemma-4-E2B-it

2026-04-025.1B dense

AA index7.8 AA estimaterank #72

Arena—

Best file & memory

Q8_04.7 GiB fileCache unknown

5.7 of 8.0 GiB+8K ctx

Prompt—Unknown

Generate—Unknown

First token · 4K prompt—Unknown

Memory budget

Weights file	4.70 GiB	unsloth/gemma-4-E2B-it-GGUF HF files
KV cache · 8K	—	Unknown
Runtime allowance	1.00 GiB	planning assumption Estimated
Recurrent state	—	Unknown
Accounted	5.70 GiB	of 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

Max context in free memory: unknown (native 128K).

Other GGUF files

Q4_K_M 2.9 GiB · cache unknown · files
Q5_K_M 3.1 GiB · cache unknown · files

Where the speeds come from

Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
Generate unknown: no measured report, and bandwidth, active size or backend calibration is missing.
First token unknown: needs a prefill value.

Caveats

KV-cache size could not be established from the config (compressed or backend-specific attention).
Fixed recurrent/convolution state is unknown and not included.
Text-only budget. A downloadable file does not prove your runtime supports this architecture.

AA 7.8 AA estimate Gemma 4 E2B (Reasoning) · Arena unknown · Model page

62 models too big for RTX 4060

GLM-5.3Q8_0 needs 748 GiB+ (KV unknown)
GLM-5.3-FlashQ8_0 needs 319 GiB+ (KV unknown)
Ling 3.1 FlashQ4_K_M needs 335 GiB+ (KV unknown)
Qwen3.8-2.4T-A95BQ8_0 needs 2,424 GiB
Qwen3.8-Flash-NextQ8_0 needs 177 GiB+ (KV unknown)
Muse Spark Open (816B-A42B)Q8_0 needs 813 GiB+ (KV unknown)
Qwen3.8-27BQ4_K_M needs 18.2 GiB
Motif 3Q4_K_M needs 185 GiB+ (KV unknown)
K2-Horizon-375B-A23BQ4_K_M needs 225 GiB
Reflection Beam 1.0 501B-A23BQ8_0 needs 498 GiB
MiniMax-M3Q4_K_M needs 245 GiB+ (KV unknown)
Nex-N2-Pro (Qwen3.5-397B-A17B based)Q4_K_M needs 227 GiB
NVIDIA Nemotron 3.5 Super VL 120B-A12BQ8_0 needs 124 GiB
Kimi K2.7 CodeQ4_K_M needs 579 GiB+ (KV unknown)
Inkling-SmallQ8_0 needs 263 GiB
K2-Horizon-MoVA-36B-A4BQ4_K_M needs 22.3 GiB+ (KV unknown)
Hy3Q4_K_M needs 174 GiB
InklingQ8_0 needs 800 GiB
Solar Open 2 (250B-A15B)Q8_0 needs 250 GiB
Ling-3.0-flash-VLQ4_K_M needs 74.8 GiB+ (KV unknown)
NVIDIA Nemotron 3 Ultra 550B-A55BQ4_K_M needs 323 GiB
Ling-3.0-flash-FinQ4_K_M needs 74.0 GiB+ (KV unknown)
G9v3-39A5B (preview)Q4_K_M needs 23.7 GiB
Qwen3.5-397B-A17BQ4_K_M needs 229 GiB
NVIDIA Nemotron 3.5 Bolt 74B-A9BQ8_0 needs 75.0 GiB
Qwen3.6-35B-A3BQ4_K_M needs 20.7 GiB
Qwen3.5-122B-A10BQ4_K_M needs 73.0 GiB
Muse Glimmer 30BQ4_K_M needs 17.3 GiB
Gemma 4 26B A4B (it)Q4_K_M needs 17.1 GiB+ (KV unknown)
Gemma 4 31B (it)Q4_K_M needs 18.6 GiB+ (KV unknown)
Mistral Medium 3.5 (128B dense)Q4_K_M needs 74.0 GiB
Gemma 4 12B (it)Q4_K_M needs 8.6 GiB+ (KV unknown)
EXAONE 4.5 33BQ4_K_M needs 21.4 GiB
Command A+ (05-2026)Q4_K_M needs 125 GiB
NVIDIA Nemotron 3.5 Lightning 30B-A3BQ4_K_M needs 25.3 GiB
IBM Granite 4.2 30BQ4_K_M needs 20.0 GiB
HyperNova 60B 2605 (compressed gpt-oss-120b)Q4_K_M needs 39.6 GiB
Nemotron Cascade 2 30B A3BQ4_K_M needs 24.4 GiB
LongCat-Flash-LiteQ4_K_M needs 39.0 GiB+ (KV unknown)
Mistral Small 4 (119B-2603)Q4_K_M needs 69.1 GiB+ (KV unknown)
Trinity-Large-ThinkingQ4_K_M needs 228 GiB
Nemotron 3 Nano Omni 30B A3B ReasoningQ4_K_M needs 24.3 GiB+ (KV unknown)
North Mini CodeQ4_K_M needs 19.4 GiB
DiffusionGemma 26B A4BQ4_K_M needs 17.2 GiB+ (KV unknown)
Qwen3-Coder-NextQ4_K_M needs 46.9 GiB
Tri-21B-ThinkQ4_K_M needs 14.8 GiB
Sarvam 105BQ4_K_M needs 61.3 GiB+ (KV unknown)
LFM2.5-24B-A2BQ4_K_M needs 15.2 GiB
MiniCPM-V 4.7 35B-A3BQ4_K_M needs 21.4 GiB
Mellum2.1-12B-A2.5B-ThinkingQ4_K_M needs 9.2 GiB
Aleph Alpha Kolibri-1Q4_K_M needs 45.9 GiB
Hy4-previewQ4_K_M needs 437 GiB+ (KV unknown)
Ling-3.0-flashQ4_K_M needs 74.0 GiB+ (KV unknown)
Poolside Laguna S 2.1Q4_K_M needs 91.4 GiB
GLM-5.2Q8_0 needs 748 GiB+ (KV unknown)
Step 3.7 FlashQ8_0 needs 197 GiB+ (KV unknown)
Qwen3.6-27BQ4_K_M needs 17.7 GiB
NVIDIA Nemotron 3 Super 120B-A12BQ8_0 needs 121 GiB
Qwen3.5-27BQ4_K_M needs 17.6 GiB
Qwen3.5-35B-A3BQ4_K_M needs 22.2 GiB
GLM-4.7-FlashQ4_K_M needs 18.6 GiB+ (KV unknown)
MiniMax-M2.7Q4_K_M needs 133 GiB

Specs & sources

Memory	8 GB GDDR6	source
Bandwidth	unknown	No official NVIDIA statement of RTX 4060 bandwidth or memory data rate found (product page, launch article, press release, VRAM article checked).
FP16	unknown	AD107 not in any NVIDIA whitepaper; only '242 AI TOPS' published.
Power	115 W	source

Measured speeds

0 reported observations

No reported observation resolved to this exact device yet.

Open in speed database Submit yours

Doesn't fit? Rent a GPU

RunpodPlaceholderCloud GPU pods you can rent for running larger models.
Vast.aiPlaceholderGPU rental marketplace with many hosts and GPU types.
LambdaPlaceholderCloud GPU instances for AI training and inference.
TensorDockPlaceholderGPU cloud marketplace with consumer and datacenter GPUs.

Sponsored placeholder links: LocalFit may earn a commission if these become affiliate links. They never influence fit results, estimates or picks.