LocalFit
Submit a benchmark

NVIDIA GeForce RTX 3060 Ti: which models fit?

RTX 4090RTX 5090RTX 3090RTX 4060 Ti (16 GB)RTX 5070 TiRadeon RX 7900 XTXM4 Max 128 GBM3 Ultra 512 GBDGX Spark (128 GB)All devices

NVIDIA · GPU

NVIDIA GeForce RTX 3060 Ti

Memory8 GB GDDR6BandwidthUnknown not published in reviewed sourcesPlanned for models8.0 GiB all VRAM usableFilesGGUF llama.cpp format

Editorial pick · 8 GB GPUsK2-Horizon-7B Q4_K_M All picks

Context

Budget = file + KV cache + 1 GiB runtime + 0.5 GiB spare. Macs: 75% of memory. Details

20 models fit RTX 3060 Ti · 8K context

Ranked by Artificial Analysis Intelligence Index, unranked models last. Arena is shown separately. How ranking works

Intelligence

Artificial Analysis Intelligence Index · top 10 models that fit RTX 3060 Ti · higher is better

  1. K2-Horizon-7B20.6
  2. K2-Horizon-3.7B15.6
  3. Qwen3.5-9B13.3
  4. Qwen3.5-4B13.1
  5. MiniCPM5-2B12.5
  6. Granite 4.2 8B11.1
  7. G9v3-3B10.8
  8. Granite 4.2 3B9.1
  9. MiniCPM5-1B8.8
  10. Nanbeige4.1-3B8.4

AA estimate

Generation speed

Tokens per second on RTX 3060 Ti, best file for each of the same models · higher is better

  1. MiniCPM5-1B~307
  2. MiniCPM5-2B~132
  3. G9v3-3B~111
  4. Granite 4.2 3B~91
  5. Nanbeige4.1-3B~84.7
  6. Qwen3.5-4B~79.1
  7. Granite 4.2 8B~66.3
  8. K2-Horizon-3.7B~65.8
  9. K2-Horizon-7B~63.4
  10. Qwen3.5-9B~53.9

MeasuredEstimated (~)

WeightsKV cacheRuntimeFreeMeasuredlinks to its sourceEstimatedshown with ~Unknownnever zero
  1. 1

    K2-Horizon-7B

    IFM/K2-Horizon-7B

    2026-09-03size unknown

    AA index20.6rank #33
    Arena—
    Best file & memory
    Q4_K_M5.2 GiB fileFits
    7.3 of 8.0 GiB8K ctx · max ≈9K
    Prompt—Unknown
    Generate~63.4 t/sEstimated
    First token · 4K prompt—Unknown

    Memory budget

    Weights file5.21 GiBIFM/K2-Horizon-7B-GGUF HF files
    KV cache · 8K1.13 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Accounted7.33 GiBof 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈9K tokens (native 512K).

    Other GGUF files

    • Q5_K_M 6.0 GiB · tight · files
    • Q8_0 8.9 GiB · too big · files
    • FP16 16.8 GiB · too big · files

    Where the speeds come from

    • Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
    • Generate estimated from median of 6 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
    • First token unknown: needs a prefill value.

    Caveats

    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 20.6 K2 Horizon 7B · Arena unknown · Model page

  2. 2

    K2-Horizon-3.7B

    IFM/K2-Horizon-3.7B

    2026-09-03size unknown

    AA index15.6rank #41
    Arena—
    Best file & memory
    Q8_05.0 GiB fileFits
    7.1 of 8.0 GiB8K ctx · max ≈11K
    Prompt—Unknown
    Generate~65.8 t/sEstimated
    First token · 4K prompt—Unknown

    Memory budget

    Weights file5.02 GiBIFM/K2-Horizon-3.7B-GGUF HF files
    KV cache · 8K1.13 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Accounted7.14 GiBof 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈11K tokens (native 512K).

    Other GGUF files

    • Q4_K_M 2.9 GiB · fits · files
    • Q5_K_M 3.4 GiB · fits · files
    • FP16 9.4 GiB · too big · files

    Where the speeds come from

    • Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
    • Generate estimated from median of 6 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
    • First token unknown: needs a prefill value.

    Caveats

    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 15.6 K2 Horizon 3.7B · Arena unknown · Model page

  3. 3

    Qwen3.5-9B

    Qwen/Qwen3.5-9B

    2026-03-029B dense

    AA index13.3 AA estimaterank #45
    Arena—
    Best file & memory
    Q5_K_M6.1 GiB fileFits · conditional
    7.4 of 8.0 GiB+8K ctx · max ≈12K
    Prompt~1,974 t/sEstimated
    Generate~53.9 t/sEstimated
    First token · 4K prompt~2.1 sEstimated

    Memory budget

    Weights file6.13 GiBunsloth/Qwen3.5-9B-GGUF HF files
    KV cache · 8K0.25 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted7.38 GiBof 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈12K tokens (native 256K).

    Other GGUF files

    • Q4_K_M 5.3 GiB · fits · files
    • Q8_0 8.9 GiB · too big · files

    Where the speeds come from

    • Prompt estimated from median of 8 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
    • Generate estimated from median of 6 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 13.3 AA estimate Qwen3.5 9B (Non-reasoning) · Arena unknown · Model page

  4. 4

    Qwen3.5-4B

    Qwen/Qwen3.5-4B

    2026-03-024B dense

    AA index13.1 AA estimaterank #48
    Arena—
    Best file & memory
    Q8_04.2 GiB fileFits · conditional
    5.4 of 8.0 GiB+8K ctx · max ≈74K
    Prompt~4,441 t/sEstimated
    Generate~79.1 t/sEstimated
    First token · 4K prompt~922 msEstimated

    Memory budget

    Weights file4.17 GiBunsloth/Qwen3.5-4B-GGUF HF files
    KV cache · 8K0.25 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted5.42 GiBof 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈74K tokens (native 256K).

    Other GGUF files

    • Q4_K_M 2.6 GiB · fits · files
    • Q5_K_M 2.9 GiB · fits · files

    Where the speeds come from

    • Prompt estimated from median of 8 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
    • Generate estimated from median of 6 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 13.1 AA estimate Qwen3.5 4B (Reasoning) · Arena unknown · Model page

  5. 5

    MiniCPM5-2B

    openbmb/MiniCPM5-2B

    2026-09-072.5B dense

    AA index12.5rank #51
    Arena—
    Best file & memory
    Q8_02.5 GiB fileFits
    3.8 of 8.0 GiB8K ctx · max ≈98K
    Prompt~7,059 t/sEstimated
    Generate~132 t/sEstimated
    First token · 4K prompt~580 msEstimated

    Memory budget

    Weights file2.50 GiBopenbmb/MiniCPM5-2B-GGUF HF files
    KV cache · 8K0.33 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Accounted3.82 GiBof 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈98K tokens (native 128K).

    Other GGUF files

    • Q4_K_M 1.5 GiB · fits · files
    • Q5_K_M 1.8 GiB · fits · files
    • FP16 4.7 GiB · fits · files

    Where the speeds come from

    • Prompt estimated from median of 8 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
    • Generate estimated from median of 6 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 12.5 MiniCPM5-2B · Arena unknown · Model page

  6. 6

    Granite 4.2 8B

    ibm-granite/granite-4.2-8b

    2026-08-258B dense

    AA index11.1rank #56
    Arena1,320
    Best file & memory
    Q4_K_M5.0 GiB fileFits
    7.2 of 8.0 GiB8K ctx · max ≈10K
    Prompt~2,221 t/sEstimated
    Generate~66.3 t/sEstimated
    First token · 4K prompt~1.8 sEstimated

    Memory budget

    Weights file4.98 GiBibm-granite/granite-4.2-8b-GGUF HF files
    KV cache · 8K1.25 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Accounted7.23 GiBof 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈10K tokens (native 128K).

    Other GGUF files

    • Q5_K_M 5.8 GiB · tight · files
    • Q8_0 8.7 GiB · too big · files
    • FP16 16.4 GiB · too big · files

    Where the speeds come from

    • Prompt estimated from median of 8 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
    • Generate estimated from median of 6 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 11.1 Granite 4.2 8B · Arena 1,320 · Model page

  7. 7

    G9v3-3B

    AI9Stars/G9v3-3B

    2026-07 (day unknown)3B dense

    AA index10.8 AA estimaterank #58
    Arena—
    Best file & memory
    Q8_03.0 GiB fileFits
    4.4 of 8.0 GiB8K ctx · max ≈70K
    Prompt~5,922 t/sEstimated
    Generate~111 t/sEstimated
    First token · 4K prompt~692 msEstimated

    Memory budget

    Weights file2.96 GiBmradermacher/G9v3-3B-GGUF HF files
    KV cache · 8K0.41 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Accounted4.37 GiBof 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈70K tokens (native 128K).

    Other GGUF files

    • Q4_K_M 1.7 GiB · fits · files
    • Q5_K_M 2.0 GiB · fits · files
    • FP16 5.6 GiB · fits · files

    Where the speeds come from

    • Prompt estimated from median of 8 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
    • Generate estimated from median of 6 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 10.8 AA estimate G9v3-3B · Arena unknown · Model page

  8. 8

    Granite 4.2 3B

    ibm-granite/granite-4.2-3b

    2026-08-253B dense

    AA index9.1rank #64
    Arena1,293
    Best file & memory
    Q8_03.6 GiB fileFits
    5.3 of 8.0 GiB8K ctx · max ≈37K
    Prompt~5,922 t/sEstimated
    Generate~91 t/sEstimated
    First token · 4K prompt~692 msEstimated

    Memory budget

    Weights file3.63 GiBibm-granite/granite-4.2-3b-GGUF HF files
    KV cache · 8K0.63 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Accounted5.25 GiBof 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈37K tokens (native 128K).

    Other GGUF files

    • Q4_K_M 2.1 GiB · fits · files
    • Q5_K_M 2.4 GiB · fits · files
    • FP16 6.8 GiB · too big · files

    Where the speeds come from

    • Prompt estimated from median of 8 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
    • Generate estimated from median of 6 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 9.1 Granite 4.2 3B · Arena 1,293 · Model page

  9. 9

    MiniCPM5-1B

    openbmb/MiniCPM5-1B

    2026-05-191.1B dense

    AA index8.8 AA estimaterank #67
    Arena—
    Best file & memory
    Q8_01.1 GiB fileFits
    2.3 of 8.0 GiB8K ctx · max ≈128K
    Prompt~16,439 t/sEstimated
    Generate~307 t/sEstimated
    First token · 4K prompt~249 msEstimated

    Memory budget

    Weights file1.07 GiBopenbmb/MiniCPM5-1B-GGUF HF files
    KV cache · 8K0.19 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Accounted2.26 GiBof 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈128K tokens (native 128K).

    Other GGUF files

    • Q4_K_M 0.6 GiB · fits · files
    • Q5_K_M 0.7 GiB · fits · files
    • FP16 2.0 GiB · fits · files

    Where the speeds come from

    • Prompt estimated from median of 8 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
    • Generate estimated from median of 6 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 8.8 AA estimate MiniCPM5-1B (Reasoning) · Arena unknown · Model page

  10. 10

    Nanbeige4.1-3B

    Nanbeige/Nanbeige4.1-3B

    2026-02 (day unknown)3B dense

    AA index8.4 AA estimaterank #69
    Arena—
    Best file & memory
    Q8_03.9 GiB fileFits
    5.4 of 8.0 GiB8K ctx · max ≈42K
    Prompt~5,922 t/sEstimated
    Generate~84.7 t/sEstimated
    First token · 4K prompt~692 msEstimated

    Memory budget

    Weights file3.90 GiBmradermacher/Nanbeige4.1-3B-GGUF HF files
    KV cache · 8K0.50 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Accounted5.40 GiBof 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈42K tokens (native 256K).

    Other GGUF files

    • Q4_K_M 2.3 GiB · fits · files
    • Q5_K_M 2.6 GiB · fits · files
    • FP16 7.3 GiB · too big · files

    Where the speeds come from

    • Prompt estimated from median of 8 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
    • Generate estimated from median of 6 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 8.4 AA estimate Nanbeige4.1-3B · Arena unknown · Model page

  11. 11

    LFM2.5-2.6B

    LiquidAI/LFM2.5-2.6B

    2026-08-042.7B dense

    AA index8.4 AA estimaterank #70
    Arena—
    Best file & memory
    Q8_02.7 GiB fileFits · conditional
    3.8 of 8.0 GiB+8K ctx · max ≈128K
    Prompt~6,604 t/sEstimated
    Generate~123 t/sEstimated
    First token · 4K prompt~620 msEstimated

    Memory budget

    Weights file2.68 GiBLiquidAI/LFM2.5-2.6B-GGUF HF files
    KV cache · 8K0.13 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted3.80 GiBof 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈128K tokens (native 128K).

    Other GGUF files

    • Q4_K_M 1.6 GiB · fits · files
    • Q5_K_M 1.8 GiB · fits · files
    • FP16 5.0 GiB · fits · files

    Where the speeds come from

    • Prompt estimated from median of 8 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
    • Generate estimated from median of 6 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 8.4 AA estimate LFM2.5-2.6B · Arena unknown · Model page

  12. 12

    Falcon-H1R-7B

    tiiuae/Falcon-H1R-7B

    2026-01-057B dense

    AA index7.8 AA estimaterank #71
    Arena—
    Best file & memory
    Q5_K_M5.0 GiB fileFits · conditional
    6.4 of 8.0 GiB+8K ctx · max ≈34K
    Prompt~2,538 t/sEstimated
    Generate~65.7 t/sEstimated
    First token · 4K prompt~1.6 sEstimated

    Memory budget

    Weights file5.02 GiBtiiuae/Falcon-H1R-7B-GGUF HF files
    KV cache · 8K0.34 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted6.36 GiBof 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈34K tokens (native 256K).

    Other GGUF files

    • Q4_K_M 4.3 GiB · fits · files
    • Q8_0 7.5 GiB · too big · files
    • FP16 14.1 GiB · too big · files

    Where the speeds come from

    • Prompt estimated from median of 8 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
    • Generate estimated from median of 6 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 7.8 AA estimate Falcon-H1R-7B · Arena unknown · Model page

  13. 13

    Step3 VL 10B

    stepfun-ai/Step3-VL-10B

    2026-01 (day unknown)10B dense

    AA index7.7 AA estimaterank #73
    Arena—
    Best file & memory
    Q4_K_M4.7 GiB fileFits
    6.8 of 8.0 GiB8K ctx · max ≈13K
    Prompt~1,777 t/sEstimated
    Generate~70.5 t/sEstimated
    First token · 4K prompt~2.3 sEstimated

    Memory budget

    Weights file4.68 GiBJamePeng2023/Step3-VL-10B-GGUF HF files
    KV cache · 8K1.13 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Accounted6.81 GiBof 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈13K tokens (native 64K).

    Other GGUF files

    • Q5_K_M 5.4 GiB · tight · files
    • Q8_0 8.1 GiB · too big · files
    • FP16 15.3 GiB · too big · files

    Where the speeds come from

    • Prompt estimated from median of 8 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
    • Generate estimated from median of 6 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 7.7 AA estimate Step3 VL 10B · Arena unknown · Model page

  14. 14

    LFM2.5-8B-A1B

    LiquidAI/LFM2.5-8B-A1B

    2026-05-288.3B · 1.5B active

    AA index7.2 AA estimaterank #74
    Arena—
    Best file & memory
    Q5_K_M5.6 GiB fileFits · conditional
    6.7 of 8.0 GiB+8K ctx · max ≈75K
    Prompt—Unknown
    Generate—Unknown
    First token · 4K prompt—Unknown

    Memory budget

    Weights file5.62 GiBLiquidAI/LFM2.5-8B-A1B-GGUF HF files
    KV cache · 8K0.09 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted6.71 GiBof 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈75K tokens (native 125K).

    Other GGUF files

    • Q4_K_M 4.8 GiB · fits · files
    • Q8_0 8.4 GiB · too big · files
    • FP16 15.8 GiB · too big · files

    Where the speeds come from

    • Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
    • Generate unknown: no measured report, and bandwidth, active size or backend calibration is missing.
    • First token unknown: needs a prefill value.

    Caveats

    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 7.2 AA estimate LFM2.5-8B-A1B · Arena unknown · Model page

  15. 15

    NVIDIA Nemotron 3 Nano 4B

    nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16

    2026-03-164B dense

    AA index4.8rank #75
    Arena—
    Best file & memory
    Q8_03.9 GiB fileFits · conditional
    5.1 of 8.0 GiB+8K ctx · max ≈164K
    Prompt~4,475 t/sEstimated
    Generate~83.7 t/sEstimated
    First token · 4K prompt~915 msEstimated

    Memory budget

    Weights file3.94 GiBunsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF HF files
    KV cache · 8K0.13 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted5.07 GiBof 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈164K tokens (native 256K).

    Other GGUF files

    • Q4_K_M 2.6 GiB · fits · files
    • Q5_K_M 2.9 GiB · fits · files

    Where the speeds come from

    • Prompt estimated from median of 8 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
    • Generate estimated from median of 6 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 4.8 NVIDIA Nemotron 3 Nano 4B · Arena unknown · Model page

  16. 16

    MiMo-V2.6-Distill-Qwen-9B

    XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B

    2026-09-229B dense

    AA index—unranked
    Arena—
    Best file & memory
    Q4_K_M5.4 GiB fileFits · conditional
    6.7 of 8.0 GiB+8K ctx · max ≈34K
    Prompt~1,974 t/sEstimated
    Generate~60.7 t/sEstimated
    First token · 4K prompt~2.1 sEstimated

    Memory budget

    Weights file5.44 GiBbartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF HF files
    KV cache · 8K0.25 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted6.69 GiBof 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈34K tokens (native 256K).

    Other GGUF files

    • Q5_K_M 6.4 GiB · tight · files
    • Q8_0 8.9 GiB · too big · files
    • FP16 16.7 GiB · too big · files

    Where the speeds come from

    • Prompt estimated from median of 8 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
    • Generate estimated from median of 6 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA index unknown · Arena unknown · Model page

  17. 17

    K2-Horizon-0.9B

    IFM/K2-Horizon-0.9B

    2026-09-03size unknown

    AA index—unranked
    Arena—
    Best file & memory
    Q8_01.1 GiB fileFits
    2.5 of 8.0 GiB8K ctx · max ≈99K
    Prompt—Unknown
    Generate~309 t/sEstimated
    First token · 4K prompt—Unknown

    Memory budget

    Weights file1.07 GiBIFM/K2-Horizon-0.9B-GGUF HF files
    KV cache · 8K0.44 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Accounted2.51 GiBof 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈99K tokens (native 128K).

    Other GGUF files

    • Q4_K_M 0.6 GiB · fits · files
    • Q5_K_M 0.7 GiB · fits · files
    • FP16 2.0 GiB · fits · files

    Where the speeds come from

    • Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
    • Generate estimated from median of 6 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
    • First token unknown: needs a prefill value.

    Caveats

    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA index unknown · Arena unknown · Model page

  18. 18

    Ornith-1.5-9B

    ornith-ai/Ornith-1.5-9B

    2026-08 (day unknown)9B dense

    AA index—unranked
    Arena—
    Best file & memory
    Q5_K_M6.2 GiB fileFits · conditional
    7.4 of 8.0 GiB+8K ctx · max ≈10K
    Prompt~1,974 t/sEstimated
    Generate~53.4 t/sEstimated
    First token · 4K prompt~2.1 sEstimated

    Memory budget

    Weights file6.19 GiBornith-ai/Ornith-1.5-9B-GGUF HF files
    KV cache · 8K0.25 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted7.44 GiBof 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈10K tokens (native 256K).

    Other GGUF files

    • Q4_K_M 5.4 GiB · fits · files
    • Q8_0 9.1 GiB · too big · files
    • FP16 16.7 GiB · too big · files

    Where the speeds come from

    • Prompt estimated from median of 8 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
    • Generate estimated from median of 6 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA index unknown · Arena unknown · Model page

  19. 19

    Qwen3.5-0.8B

    Qwen/Qwen3.5-0.8B

    2026-03-020.8B dense

    AA index—unranked
    Arena—
    Best file & memory
    Q8_00.8 GiB fileFits · conditional
    1.8 of 8.0 GiB+8K ctx · max ≈256K
    Prompt~22,206 t/sEstimated
    Generate~437 t/sEstimated
    First token · 4K prompt~184 msEstimated

    Memory budget

    Weights file0.76 GiBunsloth/Qwen3.5-0.8B-GGUF HF files
    KV cache · 8K0.09 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted1.85 GiBof 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈256K tokens (native 256K).

    Other GGUF files

    • Q4_K_M 0.5 GiB · fits · files
    • Q5_K_M 0.5 GiB · fits · files
    • FP16 1.4 GiB · fits · files

    Where the speeds come from

    • Prompt estimated from median of 8 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
    • Generate estimated from median of 6 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA index unknown · Arena unknown · Model page

  20. 20

    Qwen3.5-2B

    Qwen/Qwen3.5-2B

    2026-03-022B dense

    AA index—unranked
    Arena—
    Best file & memory
    Q8_01.9 GiB fileFits · conditional
    3.0 of 8.0 GiB+8K ctx · max ≈256K
    Prompt~8,883 t/sEstimated
    Generate~176 t/sEstimated
    First token · 4K prompt~461 msEstimated

    Memory budget

    Weights file1.87 GiBunsloth/Qwen3.5-2B-GGUF HF files
    KV cache · 8K0.09 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted2.97 GiBof 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈256K tokens (native 256K).

    Other GGUF files

    • Q4_K_M 1.2 GiB · fits · files
    • Q5_K_M 1.3 GiB · fits · files
    • FP16 3.5 GiB · fits · files

    Where the speeds come from

    • Prompt estimated from median of 8 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
    • Generate estimated from median of 6 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA index unknown · Arena unknown · Model page

May fit: KV-cache size unknown 3

Weights and runtime fit, but the cache for this attention type is not established. Ranked by AA index.

  1. Ling-3.0-tiny

    inclusionAI/Ling-3.0-tiny

    2026-08 (day unknown)7.9B · 1.3B active

    AA index11.1rank #57
    Arena—
    Best file & memory
    Q5_K_M5.2 GiB fileCache unknown
    6.2 of 8.0 GiB+8K ctx
    Prompt—Unknown
    Generate—Unknown
    First token · 4K prompt—Unknown

    Memory budget

    Weights file5.25 GiBinclusionAI/Ling-3.0-tiny-GGUF HF files
    KV cache · 8K—Unknown
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted6.25 GiBof 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

    Max context in free memory: unknown (native 128K).

    Other GGUF files

    • Q4_K_M 4.5 GiB · cache unknown · files
    • Q8_0 7.8 GiB · too big · files
    • FP16 14.7 GiB · too big · files

    Where the speeds come from

    • Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
    • Generate unknown: no measured report, and bandwidth, active size or backend calibration is missing.
    • First token unknown: needs a prefill value.

    Caveats

    • KV-cache size could not be established from the config (compressed or backend-specific attention).
    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 11.1 Ling 3.0 Tiny · Arena unknown · Model page

  2. Gemma 4 E4B (it)

    google/gemma-4-E4B-it

    2026-04-028B dense

    AA index8.9rank #66
    Arena—
    Best file & memory
    Q5_K_M5.1 GiB fileCache unknown
    6.1 of 8.0 GiB+8K ctx
    Prompt~2,221 t/sEstimated
    Generate~64.7 t/sEstimated
    First token · 4K prompt~1.8 sEstimated

    Memory budget

    Weights file5.11 GiBunsloth/gemma-4-E4B-it-GGUF HF files
    KV cache · 8K—Unknown
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted6.11 GiBof 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

    Max context in free memory: unknown (native 128K).

    Other GGUF files

    • Q4_K_M 4.6 GiB · cache unknown · files
    • Q8_0 7.6 GiB · too big · files

    Where the speeds come from

    • Prompt estimated from median of 8 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
    • Generate estimated from median of 6 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • KV-cache size could not be established from the config (compressed or backend-specific attention).
    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 8.9 Gemma 4 E4B (Reasoning) · Arena unknown · Model page

  3. Gemma 4 E2B

    google/gemma-4-E2B-it

    2026-04-025.1B dense

    AA index7.8 AA estimaterank #72
    Arena—
    Best file & memory
    Q8_04.7 GiB fileCache unknown
    5.7 of 8.0 GiB+8K ctx
    Prompt~3,483 t/sEstimated
    Generate~70.2 t/sEstimated
    First token · 4K prompt~1.2 sEstimated

    Memory budget

    Weights file4.70 GiBunsloth/gemma-4-E2B-it-GGUF HF files
    KV cache · 8K—Unknown
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted5.70 GiBof 8.0 GiB usable (100% of 8 GB) · 0.5 GiB spare margin required

    Max context in free memory: unknown (native 128K).

    Other GGUF files

    • Q4_K_M 2.9 GiB · cache unknown · files
    • Q5_K_M 3.1 GiB · cache unknown · files

    Where the speeds come from

    • Prompt estimated from median of 8 measured prefill points for this device (llama.cpp CUDA), rescaled to this model.
    • Generate estimated from median of 6 measured decode points for this device (llama.cpp CUDA), rescaled to this model.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • KV-cache size could not be established from the config (compressed or backend-specific attention).
    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 7.8 AA estimate Gemma 4 E2B (Reasoning) · Arena unknown · Model page

62 models too big for RTX 3060 Ti

Specs & sources

Memory8 GB GDDR6source
BandwidthunknownNo official NVIDIA statement of RTX 3060 Ti bandwidth/data rate found (product page, launch article, press release checked).
FP16unknownGA104 RTX 3060 Ti is not in any NVIDIA whitepaper.
Power200 Wsource

Doesn't fit? Rent a GPU

  • RunpodPlaceholderCloud GPU pods you can rent for running larger models.
  • Vast.aiPlaceholderGPU rental marketplace with many hosts and GPU types.
  • LambdaPlaceholderCloud GPU instances for AI training and inference.
  • TensorDockPlaceholderGPU cloud marketplace with consumer and datacenter GPUs.

Sponsored placeholder links: LocalFit may earn a commission if these become affiliate links. They never influence fit results, estimates or picks.