LocalFit
Submit a benchmark

AMD Radeon RX 9070 GRE: which models fit?

RTX 4090RTX 5090RTX 3090RTX 4060 Ti (16 GB)RTX 5070 TiRadeon RX 7900 XTXM4 Max 128 GBM3 Ultra 512 GBDGX Spark (128 GB)All devices

AMD · GPU

AMD Radeon RX 9070 GRE

Memory12 GB GDDR6Bandwidth432 GB/s vendor specPlanned for models12.0 GiB all VRAM usableFilesGGUF llama.cpp format

Editorial pick · 12 GB GPUsK2-Horizon-7B Q8_0 All picks

Context

Budget = file + KV cache + 1 GiB runtime + 0.5 GiB spare. Macs: 75% of memory. Details

21 models fit Radeon RX 9070 GRE · 8K context

Ranked by Artificial Analysis Intelligence Index, unranked models last. Arena is shown separately. How ranking works

Intelligence

Artificial Analysis Intelligence Index · top 10 models that fit Radeon RX 9070 GRE · higher is better

  1. K2-Horizon-7B20.6
  2. K2-Horizon-3.7B15.6
  3. Qwen3.5-9B13.3
  4. Qwen3.5-4B13.1
  5. MiniCPM5-2B12.5
  6. Granite 4.2 8B11.1
  7. G9v3-3B10.8
  8. Granite 4.2 3B9.1
  9. MiniCPM5-1B8.8
  10. Nanbeige4.1-3B8.4

AA estimate

Generation speed

Tokens per second on Radeon RX 9070 GRE, best file for each of the same models · higher is better

  1. MiniCPM5-1B~209
  2. MiniCPM5-2B~90.1
  3. G9v3-3B~75.9
  4. Granite 4.2 3B~62
  5. Nanbeige4.1-3B~57.7
  6. Qwen3.5-4B~53.8
  7. K2-Horizon-3.7B~44.8
  8. Granite 4.2 8B~25.8
  9. Qwen3.5-9B~25.3
  10. K2-Horizon-7B~25.2

MeasuredEstimated (~)

WeightsKV cacheRuntimeFreeMeasuredlinks to its sourceEstimatedshown with ~Unknownnever zero
  1. 1

    K2-Horizon-7B

    IFM/K2-Horizon-7B

    2026-09-03size unknown

    AA index20.6rank #33
    Arena—
    Best file & memory
    Q8_08.9 GiB fileFits
    11.0 of 12.0 GiB8K ctx · max ≈11K
    Prompt—Unknown
    Generate~25.2 t/sEstimated
    First token · 4K prompt—Unknown

    Memory budget

    Weights file8.92 GiBIFM/K2-Horizon-7B-GGUF HF files
    KV cache · 8K1.13 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Accounted11.04 GiBof 12.0 GiB usable (100% of 12 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈11K tokens (native 512K).

    Other GGUF files

    • Q4_K_M 5.2 GiB · fits · files
    • Q5_K_M 6.0 GiB · fits · files
    • FP16 16.8 GiB · too big · files

    Where the speeds come from

    • Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
    • Generate estimated from llama.cpp ROCm median efficiency 0.5586 (n=117) × memory bandwidth ÷ 8.9 GiB read per token.
    • First token unknown: needs a prefill value.

    Caveats

    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 20.6 K2 Horizon 7B · Arena unknown · Model page

  2. 2

    K2-Horizon-3.7B

    IFM/K2-Horizon-3.7B

    2026-09-03size unknown

    AA index15.6rank #41
    Arena—
    Best file & memory
    Q8_05.0 GiB fileFits
    7.1 of 12.0 GiB8K ctx · max ≈39K
    Prompt—Unknown
    Generate~44.8 t/sEstimated
    First token · 4K prompt—Unknown

    Memory budget

    Weights file5.02 GiBIFM/K2-Horizon-3.7B-GGUF HF files
    KV cache · 8K1.13 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Accounted7.14 GiBof 12.0 GiB usable (100% of 12 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈39K tokens (native 512K).

    Other GGUF files

    • Q4_K_M 2.9 GiB · fits · files
    • Q5_K_M 3.4 GiB · fits · files
    • FP16 9.4 GiB · tight · files

    Where the speeds come from

    • Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
    • Generate estimated from llama.cpp ROCm median efficiency 0.5586 (n=117) × memory bandwidth ÷ 5.0 GiB read per token.
    • First token unknown: needs a prefill value.

    Caveats

    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 15.6 K2 Horizon 3.7B · Arena unknown · Model page

  3. 3

    Qwen3.5-9B

    Qwen/Qwen3.5-9B

    2026-03-029B dense

    AA index13.3 AA estimaterank #45
    Arena—
    Best file & memory
    Q8_08.9 GiB fileFits · conditional
    10.1 of 12.0 GiB+8K ctx · max ≈52K
    Prompt~2,640 t/sEstimated
    Generate~25.3 t/sEstimated
    First token · 4K prompt~1.6 sEstimated

    Memory budget

    Weights file8.87 GiBunsloth/Qwen3.5-9B-GGUF HF files
    KV cache · 8K0.25 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted10.12 GiBof 12.0 GiB usable (100% of 12 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈52K tokens (native 256K).

    Other GGUF files

    • Q4_K_M 5.3 GiB · fits · files
    • Q5_K_M 6.1 GiB · fits · files

    Where the speeds come from

    • Prompt estimated from llama.cpp ROCm median efficiency 0.3468 (n=116) × vendor FP16 TFLOPS ÷ (2 × active params).
    • Generate estimated from llama.cpp ROCm median efficiency 0.5586 (n=117) × memory bandwidth ÷ 8.9 GiB read per token.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 13.3 AA estimate Qwen3.5 9B (Non-reasoning) · Arena unknown · Model page

  4. 4

    Qwen3.5-4B

    Qwen/Qwen3.5-4B

    2026-03-024B dense

    AA index13.1 AA estimaterank #48
    Arena—
    Best file & memory
    Q8_04.2 GiB fileFits · conditional
    5.4 of 12.0 GiB+8K ctx · max ≈202K
    Prompt~5,939 t/sEstimated
    Generate~53.8 t/sEstimated
    First token · 4K prompt~690 msEstimated

    Memory budget

    Weights file4.17 GiBunsloth/Qwen3.5-4B-GGUF HF files
    KV cache · 8K0.25 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted5.42 GiBof 12.0 GiB usable (100% of 12 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈202K tokens (native 256K).

    Other GGUF files

    • Q4_K_M 2.6 GiB · fits · files
    • Q5_K_M 2.9 GiB · fits · files

    Where the speeds come from

    • Prompt estimated from llama.cpp ROCm median efficiency 0.3468 (n=116) × vendor FP16 TFLOPS ÷ (2 × active params).
    • Generate estimated from llama.cpp ROCm median efficiency 0.5586 (n=117) × memory bandwidth ÷ 4.2 GiB read per token.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 13.1 AA estimate Qwen3.5 4B (Reasoning) · Arena unknown · Model page

  5. 5

    MiniCPM5-2B

    openbmb/MiniCPM5-2B

    2026-09-072.5B dense

    AA index12.5rank #51
    Arena—
    Best file & memory
    Q8_02.5 GiB fileFits
    3.8 of 12.0 GiB8K ctx · max ≈128K
    Prompt~9,439 t/sEstimated
    Generate~90.1 t/sEstimated
    First token · 4K prompt~434 msEstimated

    Memory budget

    Weights file2.50 GiBopenbmb/MiniCPM5-2B-GGUF HF files
    KV cache · 8K0.33 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Accounted3.82 GiBof 12.0 GiB usable (100% of 12 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈128K tokens (native 128K).

    Other GGUF files

    • Q4_K_M 1.5 GiB · fits · files
    • Q5_K_M 1.8 GiB · fits · files
    • FP16 4.7 GiB · fits · files

    Where the speeds come from

    • Prompt estimated from llama.cpp ROCm median efficiency 0.3468 (n=116) × vendor FP16 TFLOPS ÷ (2 × active params).
    • Generate estimated from llama.cpp ROCm median efficiency 0.5586 (n=117) × memory bandwidth ÷ 2.5 GiB read per token.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 12.5 MiniCPM5-2B · Arena unknown · Model page

  6. 6

    Granite 4.2 8B

    ibm-granite/granite-4.2-8b

    2026-08-258B dense

    AA index11.1rank #56
    Arena1,320
    Best file & memory
    Q8_08.7 GiB fileFits
    11.0 of 12.0 GiB8K ctx · max ≈11K
    Prompt~2,969 t/sEstimated
    Generate~25.8 t/sEstimated
    First token · 4K prompt~1.4 sEstimated

    Memory budget

    Weights file8.70 GiBibm-granite/granite-4.2-8b-GGUF HF files
    KV cache · 8K1.25 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Accounted10.95 GiBof 12.0 GiB usable (100% of 12 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈11K tokens (native 128K).

    Other GGUF files

    • Q4_K_M 5.0 GiB · fits · files
    • Q5_K_M 5.8 GiB · fits · files
    • FP16 16.4 GiB · too big · files

    Where the speeds come from

    • Prompt estimated from llama.cpp ROCm median efficiency 0.3468 (n=116) × vendor FP16 TFLOPS ÷ (2 × active params).
    • Generate estimated from llama.cpp ROCm median efficiency 0.5586 (n=117) × memory bandwidth ÷ 8.7 GiB read per token.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 11.1 Granite 4.2 8B · Arena 1,320 · Model page

  7. 7

    G9v3-3B

    AI9Stars/G9v3-3B

    2026-07 (day unknown)3B dense

    AA index10.8 AA estimaterank #58
    Arena—
    Best file & memory
    Q8_03.0 GiB fileFits
    4.4 of 12.0 GiB8K ctx · max ≈128K
    Prompt~7,919 t/sEstimated
    Generate~75.9 t/sEstimated
    First token · 4K prompt~517 msEstimated

    Memory budget

    Weights file2.96 GiBmradermacher/G9v3-3B-GGUF HF files
    KV cache · 8K0.41 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Accounted4.37 GiBof 12.0 GiB usable (100% of 12 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈128K tokens (native 128K).

    Other GGUF files

    • Q4_K_M 1.7 GiB · fits · files
    • Q5_K_M 2.0 GiB · fits · files
    • FP16 5.6 GiB · fits · files

    Where the speeds come from

    • Prompt estimated from llama.cpp ROCm median efficiency 0.3468 (n=116) × vendor FP16 TFLOPS ÷ (2 × active params).
    • Generate estimated from llama.cpp ROCm median efficiency 0.5586 (n=117) × memory bandwidth ÷ 3.0 GiB read per token.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 10.8 AA estimate G9v3-3B · Arena unknown · Model page

  8. 8

    Granite 4.2 3B

    ibm-granite/granite-4.2-3b

    2026-08-253B dense

    AA index9.1rank #64
    Arena1,293
    Best file & memory
    Q8_03.6 GiB fileFits
    5.3 of 12.0 GiB8K ctx · max ≈88K
    Prompt~7,919 t/sEstimated
    Generate~62 t/sEstimated
    First token · 4K prompt~517 msEstimated

    Memory budget

    Weights file3.63 GiBibm-granite/granite-4.2-3b-GGUF HF files
    KV cache · 8K0.63 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Accounted5.25 GiBof 12.0 GiB usable (100% of 12 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈88K tokens (native 128K).

    Other GGUF files

    • Q4_K_M 2.1 GiB · fits · files
    • Q5_K_M 2.4 GiB · fits · files
    • FP16 6.8 GiB · fits · files

    Where the speeds come from

    • Prompt estimated from llama.cpp ROCm median efficiency 0.3468 (n=116) × vendor FP16 TFLOPS ÷ (2 × active params).
    • Generate estimated from llama.cpp ROCm median efficiency 0.5586 (n=117) × memory bandwidth ÷ 3.6 GiB read per token.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 9.1 Granite 4.2 3B · Arena 1,293 · Model page

  9. 9

    MiniCPM5-1B

    openbmb/MiniCPM5-1B

    2026-05-191.1B dense

    AA index8.8 AA estimaterank #67
    Arena—
    Best file & memory
    Q8_01.1 GiB fileFits
    2.3 of 12.0 GiB8K ctx · max ≈128K
    Prompt~21,983 t/sEstimated
    Generate~209 t/sEstimated
    First token · 4K prompt~186 msEstimated

    Memory budget

    Weights file1.07 GiBopenbmb/MiniCPM5-1B-GGUF HF files
    KV cache · 8K0.19 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Accounted2.26 GiBof 12.0 GiB usable (100% of 12 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈128K tokens (native 128K).

    Other GGUF files

    • Q4_K_M 0.6 GiB · fits · files
    • Q5_K_M 0.7 GiB · fits · files
    • FP16 2.0 GiB · fits · files

    Where the speeds come from

    • Prompt estimated from llama.cpp ROCm median efficiency 0.3468 (n=116) × vendor FP16 TFLOPS ÷ (2 × active params).
    • Generate estimated from llama.cpp ROCm median efficiency 0.5586 (n=117) × memory bandwidth ÷ 1.1 GiB read per token.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 8.8 AA estimate MiniCPM5-1B (Reasoning) · Arena unknown · Model page

  10. 10

    Nanbeige4.1-3B

    Nanbeige/Nanbeige4.1-3B

    2026-02 (day unknown)3B dense

    AA index8.4 AA estimaterank #69
    Arena—
    Best file & memory
    Q8_03.9 GiB fileFits
    5.4 of 12.0 GiB8K ctx · max ≈106K
    Prompt~7,919 t/sEstimated
    Generate~57.7 t/sEstimated
    First token · 4K prompt~517 msEstimated

    Memory budget

    Weights file3.90 GiBmradermacher/Nanbeige4.1-3B-GGUF HF files
    KV cache · 8K0.50 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Accounted5.40 GiBof 12.0 GiB usable (100% of 12 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈106K tokens (native 256K).

    Other GGUF files

    • Q4_K_M 2.3 GiB · fits · files
    • Q5_K_M 2.6 GiB · fits · files
    • FP16 7.3 GiB · fits · files

    Where the speeds come from

    • Prompt estimated from llama.cpp ROCm median efficiency 0.3468 (n=116) × vendor FP16 TFLOPS ÷ (2 × active params).
    • Generate estimated from llama.cpp ROCm median efficiency 0.5586 (n=117) × memory bandwidth ÷ 3.9 GiB read per token.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 8.4 AA estimate Nanbeige4.1-3B · Arena unknown · Model page

  11. 11

    LFM2.5-2.6B

    LiquidAI/LFM2.5-2.6B

    2026-08-042.7B dense

    AA index8.4 AA estimaterank #70
    Arena—
    Best file & memory
    Q8_02.7 GiB fileFits · conditional
    3.8 of 12.0 GiB+8K ctx · max ≈128K
    Prompt~8,831 t/sEstimated
    Generate~83.9 t/sEstimated
    First token · 4K prompt~464 msEstimated

    Memory budget

    Weights file2.68 GiBLiquidAI/LFM2.5-2.6B-GGUF HF files
    KV cache · 8K0.13 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted3.80 GiBof 12.0 GiB usable (100% of 12 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈128K tokens (native 128K).

    Other GGUF files

    • Q4_K_M 1.6 GiB · fits · files
    • Q5_K_M 1.8 GiB · fits · files
    • FP16 5.0 GiB · fits · files

    Where the speeds come from

    • Prompt estimated from llama.cpp ROCm median efficiency 0.3468 (n=116) × vendor FP16 TFLOPS ÷ (2 × active params).
    • Generate estimated from llama.cpp ROCm median efficiency 0.5586 (n=117) × memory bandwidth ÷ 2.7 GiB read per token.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 8.4 AA estimate LFM2.5-2.6B · Arena unknown · Model page

  12. 12

    Falcon-H1R-7B

    tiiuae/Falcon-H1R-7B

    2026-01-057B dense

    AA index7.8 AA estimaterank #71
    Arena—
    Best file & memory
    Q8_07.5 GiB fileFits · conditional
    8.9 of 12.0 GiB+8K ctx · max ≈69K
    Prompt~3,394 t/sEstimated
    Generate~29.9 t/sEstimated
    First token · 4K prompt~1.2 sEstimated

    Memory budget

    Weights file7.51 GiBtiiuae/Falcon-H1R-7B-GGUF HF files
    KV cache · 8K0.34 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted8.86 GiBof 12.0 GiB usable (100% of 12 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈69K tokens (native 256K).

    Other GGUF files

    • Q4_K_M 4.3 GiB · fits · files
    • Q5_K_M 5.0 GiB · fits · files
    • FP16 14.1 GiB · too big · files

    Where the speeds come from

    • Prompt estimated from llama.cpp ROCm median efficiency 0.3468 (n=116) × vendor FP16 TFLOPS ÷ (2 × active params).
    • Generate estimated from llama.cpp ROCm median efficiency 0.5586 (n=117) × memory bandwidth ÷ 7.5 GiB read per token.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 7.8 AA estimate Falcon-H1R-7B · Arena unknown · Model page

  13. 13

    Step3 VL 10B

    stepfun-ai/Step3-VL-10B

    2026-01 (day unknown)10B dense

    AA index7.7 AA estimaterank #73
    Arena—
    Best file & memory
    Q8_08.1 GiB fileFits
    10.2 of 12.0 GiB8K ctx · max ≈17K
    Prompt~2,376 t/sEstimated
    Generate~27.7 t/sEstimated
    First token · 4K prompt~1.7 sEstimated

    Memory budget

    Weights file8.11 GiBJamePeng2023/Step3-VL-10B-GGUF HF files
    KV cache · 8K1.13 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Accounted10.24 GiBof 12.0 GiB usable (100% of 12 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈17K tokens (native 64K).

    Other GGUF files

    • Q4_K_M 4.7 GiB · fits · files
    • Q5_K_M 5.4 GiB · fits · files
    • FP16 15.3 GiB · too big · files

    Where the speeds come from

    • Prompt estimated from llama.cpp ROCm median efficiency 0.3468 (n=116) × vendor FP16 TFLOPS ÷ (2 × active params).
    • Generate estimated from llama.cpp ROCm median efficiency 0.5586 (n=117) × memory bandwidth ÷ 8.1 GiB read per token.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 7.7 AA estimate Step3 VL 10B · Arena unknown · Model page

  14. 14

    LFM2.5-8B-A1B

    LiquidAI/LFM2.5-8B-A1B

    2026-05-288.3B · 1.5B active

    AA index7.2 AA estimaterank #74
    Arena—
    Best file & memory
    Q8_08.4 GiB fileFits · conditional
    9.5 of 12.0 GiB+8K ctx · max ≈125K
    Prompt~4,781 t/sEstimated
    Generate~148 t/sEstimated
    First token · 4K prompt~857 msEstimated

    Memory budget

    Weights file8.39 GiBLiquidAI/LFM2.5-8B-A1B-GGUF HF files
    KV cache · 8K0.09 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted9.49 GiBof 12.0 GiB usable (100% of 12 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈125K tokens (native 125K).

    Other GGUF files

    • Q4_K_M 4.8 GiB · fits · files
    • Q5_K_M 5.6 GiB · fits · files
    • FP16 15.8 GiB · too big · files

    Where the speeds come from

    • Prompt estimated from llama.cpp ROCm median efficiency 0.1047 (n=7) × vendor FP16 TFLOPS ÷ (2 × active params).
    • Generate estimated from llama.cpp ROCm median efficiency 0.5564 (n=119) × memory bandwidth ÷ 1.5 GiB read per token.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 7.2 AA estimate LFM2.5-8B-A1B · Arena unknown · Model page

  15. 15

    NVIDIA Nemotron 3 Nano 4B

    nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16

    2026-03-164B dense

    AA index4.8rank #75
    Arena—
    Best file & memory
    Q8_03.9 GiB fileFits · conditional
    5.1 of 12.0 GiB+8K ctx · max ≈256K
    Prompt~5,984 t/sEstimated
    Generate~57 t/sEstimated
    First token · 4K prompt~685 msEstimated

    Memory budget

    Weights file3.94 GiBunsloth/NVIDIA-Nemotron-3-Nano-4B-GGUF HF files
    KV cache · 8K0.13 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted5.07 GiBof 12.0 GiB usable (100% of 12 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈256K tokens (native 256K).

    Other GGUF files

    • Q4_K_M 2.6 GiB · fits · files
    • Q5_K_M 2.9 GiB · fits · files

    Where the speeds come from

    • Prompt estimated from llama.cpp ROCm median efficiency 0.3468 (n=116) × vendor FP16 TFLOPS ÷ (2 × active params).
    • Generate estimated from llama.cpp ROCm median efficiency 0.5586 (n=117) × memory bandwidth ÷ 3.9 GiB read per token.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 4.8 NVIDIA Nemotron 3 Nano 4B · Arena unknown · Model page

  16. 16

    Mellum2.1-12B-A2.5B-Thinking

    JetBrains/Mellum2.1-12B-A2.5B-Thinking

    2026-10-0812B · 2.5B active

    AA index—unranked
    Arena—
    Best file & memory
    Q5_K_M9.3 GiB fileFits
    10.4 of 12.0 GiB8K ctx · max ≈86K
    Prompt~2,869 t/sEstimated
    Generate~116 t/sEstimated
    First token · 4K prompt~1.4 sEstimated

    Memory budget

    Weights file9.28 GiBbartowski/Mellum2.1-12B-A2.5B-Thinking-GGUF HF files
    KV cache · 8K0.15 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Accounted10.43 GiBof 12.0 GiB usable (100% of 12 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈86K tokens (native 128K).

    Other GGUF files

    • Q4_K_M 7.5 GiB · fits · files
    • Q8_0 12.0 GiB · too big · files

    Where the speeds come from

    • Prompt estimated from llama.cpp ROCm median efficiency 0.1047 (n=7) × vendor FP16 TFLOPS ÷ (2 × active params).
    • Generate estimated from llama.cpp ROCm median efficiency 0.5564 (n=119) × memory bandwidth ÷ 1.9 GiB read per token.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA index unknown · Arena unknown · Model page

  17. 17

    MiMo-V2.6-Distill-Qwen-9B

    XiaomiMiMo/MiMo-V2.6-Distill-Qwen-9B

    2026-09-229B dense

    AA index—unranked
    Arena—
    Best file & memory
    Q8_08.9 GiB fileFits · conditional
    10.1 of 12.0 GiB+8K ctx · max ≈52K
    Prompt~2,640 t/sEstimated
    Generate~25.3 t/sEstimated
    First token · 4K prompt~1.6 sEstimated

    Memory budget

    Weights file8.89 GiBbartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF HF files
    KV cache · 8K0.25 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted10.14 GiBof 12.0 GiB usable (100% of 12 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈52K tokens (native 256K).

    Other GGUF files

    • Q4_K_M 5.4 GiB · fits · files
    • Q5_K_M 6.4 GiB · fits · files
    • FP16 16.7 GiB · too big · files

    Where the speeds come from

    • Prompt estimated from llama.cpp ROCm median efficiency 0.3468 (n=116) × vendor FP16 TFLOPS ÷ (2 × active params).
    • Generate estimated from llama.cpp ROCm median efficiency 0.5586 (n=117) × memory bandwidth ÷ 8.9 GiB read per token.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA index unknown · Arena unknown · Model page

  18. 18

    K2-Horizon-0.9B

    IFM/K2-Horizon-0.9B

    2026-09-03size unknown

    AA index—unranked
    Arena—
    Best file & memory
    Q8_01.1 GiB fileFits
    2.5 of 12.0 GiB8K ctx · max ≈128K
    Prompt—Unknown
    Generate~210 t/sEstimated
    First token · 4K prompt—Unknown

    Memory budget

    Weights file1.07 GiBIFM/K2-Horizon-0.9B-GGUF HF files
    KV cache · 8K0.44 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Accounted2.51 GiBof 12.0 GiB usable (100% of 12 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈128K tokens (native 128K).

    Other GGUF files

    • Q4_K_M 0.6 GiB · fits · files
    • Q5_K_M 0.7 GiB · fits · files
    • FP16 2.0 GiB · fits · files

    Where the speeds come from

    • Prompt unknown: no measured report, and no comparable vendor FP16 figure or device calibration.
    • Generate estimated from llama.cpp ROCm median efficiency 0.5586 (n=117) × memory bandwidth ÷ 1.1 GiB read per token.
    • First token unknown: needs a prefill value.

    Caveats

    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA index unknown · Arena unknown · Model page

  19. 19

    Ornith-1.5-9B

    ornith-ai/Ornith-1.5-9B

    2026-08 (day unknown)9B dense

    AA index—unranked
    Arena—
    Best file & memory
    Q8_09.1 GiB fileFits · conditional
    10.4 of 12.0 GiB+8K ctx · max ≈44K
    Prompt~2,640 t/sEstimated
    Generate~24.7 t/sEstimated
    First token · 4K prompt~1.6 sEstimated

    Memory budget

    Weights file9.11 GiBornith-ai/Ornith-1.5-9B-GGUF HF files
    KV cache · 8K0.25 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted10.36 GiBof 12.0 GiB usable (100% of 12 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈44K tokens (native 256K).

    Other GGUF files

    • Q4_K_M 5.4 GiB · fits · files
    • Q5_K_M 6.2 GiB · fits · files
    • FP16 16.7 GiB · too big · files

    Where the speeds come from

    • Prompt estimated from llama.cpp ROCm median efficiency 0.3468 (n=116) × vendor FP16 TFLOPS ÷ (2 × active params).
    • Generate estimated from llama.cpp ROCm median efficiency 0.5586 (n=117) × memory bandwidth ÷ 9.1 GiB read per token.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA index unknown · Arena unknown · Model page

  20. 20

    Qwen3.5-0.8B

    Qwen/Qwen3.5-0.8B

    2026-03-020.8B dense

    AA index—unranked
    Arena—
    Best file & memory
    Q8_00.8 GiB fileFits · conditional
    1.8 of 12.0 GiB+8K ctx · max ≈256K
    Prompt~29,695 t/sEstimated
    Generate~297 t/sEstimated
    First token · 4K prompt~138 msEstimated

    Memory budget

    Weights file0.76 GiBunsloth/Qwen3.5-0.8B-GGUF HF files
    KV cache · 8K0.09 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted1.85 GiBof 12.0 GiB usable (100% of 12 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈256K tokens (native 256K).

    Other GGUF files

    • Q4_K_M 0.5 GiB · fits · files
    • Q5_K_M 0.5 GiB · fits · files
    • FP16 1.4 GiB · fits · files

    Where the speeds come from

    • Prompt estimated from llama.cpp ROCm median efficiency 0.3468 (n=116) × vendor FP16 TFLOPS ÷ (2 × active params).
    • Generate estimated from llama.cpp ROCm median efficiency 0.5586 (n=117) × memory bandwidth ÷ 0.8 GiB read per token.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA index unknown · Arena unknown · Model page

  21. 21

    Qwen3.5-2B

    Qwen/Qwen3.5-2B

    2026-03-022B dense

    AA index—unranked
    Arena—
    Best file & memory
    Q8_01.9 GiB fileFits · conditional
    3.0 of 12.0 GiB+8K ctx · max ≈256K
    Prompt~11,878 t/sEstimated
    Generate~120 t/sEstimated
    First token · 4K prompt~345 msEstimated

    Memory budget

    Weights file1.87 GiBunsloth/Qwen3.5-2B-GGUF HF files
    KV cache · 8K0.09 GiBFP16 cache from config.json Estimated
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted2.97 GiBof 12.0 GiB usable (100% of 12 GB) · 0.5 GiB spare margin required

    Max context in free memory: ≈256K tokens (native 256K).

    Other GGUF files

    • Q4_K_M 1.2 GiB · fits · files
    • Q5_K_M 1.3 GiB · fits · files
    • FP16 3.5 GiB · fits · files

    Where the speeds come from

    • Prompt estimated from llama.cpp ROCm median efficiency 0.3468 (n=116) × vendor FP16 TFLOPS ÷ (2 × active params).
    • Generate estimated from llama.cpp ROCm median efficiency 0.5586 (n=117) × memory bandwidth ÷ 1.9 GiB read per token.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA index unknown · Arena unknown · Model page

May fit: KV-cache size unknown 4

Weights and runtime fit, but the cache for this attention type is not established. Ranked by AA index.

  1. Gemma 4 12B (it)

    google/gemma-4-12B-it

    2026-06-0312B dense

    AA index14.2 AA estimaterank #44
    Arena—
    Best file & memory
    Q5_K_M8.2 GiB fileCache unknown
    9.2 of 12.0 GiB+8K ctx
    Prompt~1,988 t/sEstimated
    Generate~27.5 t/sEstimated
    First token · 4K prompt~2.1 sEstimated

    Memory budget

    Weights file8.17 GiBbartowski/gemma-4-12B-it-GGUF HF files
    KV cache · 8K—Unknown
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted9.17 GiBof 12.0 GiB usable (100% of 12 GB) · 0.5 GiB spare margin required

    Max context in free memory: unknown (native 256K).

    Other GGUF files

    • Q4_K_M 7.1 GiB · cache unknown · files
    • Q8_0 11.8 GiB · too big · files

    Where the speeds come from

    • Prompt estimated from llama.cpp ROCm median efficiency 0.3468 (n=116) × vendor FP16 TFLOPS ÷ (2 × active params).
    • Generate estimated from llama.cpp ROCm median efficiency 0.5586 (n=117) × memory bandwidth ÷ 8.2 GiB read per token.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • KV-cache size could not be established from the config (compressed or backend-specific attention).
    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 14.2 AA estimate Gemma 4 12B (Reasoning) · Arena unknown · Model page

  2. Ling-3.0-tiny

    inclusionAI/Ling-3.0-tiny

    2026-08 (day unknown)7.9B · 1.3B active

    AA index11.1rank #57
    Arena—
    Best file & memory
    Q8_07.8 GiB fileCache unknown
    8.8 of 12.0 GiB+8K ctx
    Prompt~5,517 t/sEstimated
    Generate~174 t/sEstimated
    First token · 4K prompt~742 msEstimated

    Memory budget

    Weights file7.83 GiBinclusionAI/Ling-3.0-tiny-GGUF HF files
    KV cache · 8K—Unknown
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted8.83 GiBof 12.0 GiB usable (100% of 12 GB) · 0.5 GiB spare margin required

    Max context in free memory: unknown (native 128K).

    Other GGUF files

    • Q4_K_M 4.5 GiB · cache unknown · files
    • Q5_K_M 5.2 GiB · cache unknown · files
    • FP16 14.7 GiB · too big · files

    Where the speeds come from

    • Prompt estimated from llama.cpp ROCm median efficiency 0.1047 (n=7) × vendor FP16 TFLOPS ÷ (2 × active params).
    • Generate estimated from llama.cpp ROCm median efficiency 0.5564 (n=119) × memory bandwidth ÷ 1.3 GiB read per token.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • KV-cache size could not be established from the config (compressed or backend-specific attention).
    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 11.1 Ling 3.0 Tiny · Arena unknown · Model page

  3. Gemma 4 E4B (it)

    google/gemma-4-E4B-it

    2026-04-028B dense

    AA index8.9rank #66
    Arena—
    Best file & memory
    Q8_07.6 GiB fileCache unknown
    8.6 of 12.0 GiB+8K ctx
    Prompt~2,969 t/sEstimated
    Generate~29.5 t/sEstimated
    First token · 4K prompt~1.4 sEstimated

    Memory budget

    Weights file7.63 GiBunsloth/gemma-4-E4B-it-GGUF HF files
    KV cache · 8K—Unknown
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted8.63 GiBof 12.0 GiB usable (100% of 12 GB) · 0.5 GiB spare margin required

    Max context in free memory: unknown (native 128K).

    Other GGUF files

    • Q4_K_M 4.6 GiB · cache unknown · files
    • Q5_K_M 5.1 GiB · cache unknown · files

    Where the speeds come from

    • Prompt estimated from llama.cpp ROCm median efficiency 0.3468 (n=116) × vendor FP16 TFLOPS ÷ (2 × active params).
    • Generate estimated from llama.cpp ROCm median efficiency 0.5586 (n=117) × memory bandwidth ÷ 7.6 GiB read per token.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • KV-cache size could not be established from the config (compressed or backend-specific attention).
    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 8.9 Gemma 4 E4B (Reasoning) · Arena unknown · Model page

  4. Gemma 4 E2B

    google/gemma-4-E2B-it

    2026-04-025.1B dense

    AA index7.8 AA estimaterank #72
    Arena—
    Best file & memory
    Q8_04.7 GiB fileCache unknown
    5.7 of 12.0 GiB+8K ctx
    Prompt~4,658 t/sEstimated
    Generate~47.8 t/sEstimated
    First token · 4K prompt~879 msEstimated

    Memory budget

    Weights file4.70 GiBunsloth/gemma-4-E2B-it-GGUF HF files
    KV cache · 8K—Unknown
    Runtime allowance1.00 GiBplanning assumption Estimated
    Recurrent state—Unknown
    Accounted5.70 GiBof 12.0 GiB usable (100% of 12 GB) · 0.5 GiB spare margin required

    Max context in free memory: unknown (native 128K).

    Other GGUF files

    • Q4_K_M 2.9 GiB · cache unknown · files
    • Q5_K_M 3.1 GiB · cache unknown · files

    Where the speeds come from

    • Prompt estimated from llama.cpp ROCm median efficiency 0.3468 (n=116) × vendor FP16 TFLOPS ÷ (2 × active params).
    • Generate estimated from llama.cpp ROCm median efficiency 0.5586 (n=117) × memory bandwidth ÷ 4.7 GiB read per token.
    • First token 4K-token prompt ÷ estimated prefill speed. Excludes model load and queueing.

    Caveats

    • KV-cache size could not be established from the config (compressed or backend-specific attention).
    • Fixed recurrent/convolution state is unknown and not included.
    • Text-only budget. A downloadable file does not prove your runtime supports this architecture.

    AA 7.8 AA estimate Gemma 4 E2B (Reasoning) · Arena unknown · Model page

60 models too big for Radeon RX 9070 GRE

Specs & sources

Memory12 GB GDDR6source
Bandwidth432 GB/ssource
FP16137 TFLOPSsource
Power220 Wsource

FP16: Peak Half Precision (FP16 Matrix) Performance (dense); accumulation precision unspecified. Power: Typical Board Power (Desktop).

Doesn't fit? Rent a GPU

  • RunpodPlaceholderCloud GPU pods you can rent for running larger models.
  • Vast.aiPlaceholderGPU rental marketplace with many hosts and GPU types.
  • LambdaPlaceholderCloud GPU instances for AI training and inference.
  • TensorDockPlaceholderGPU cloud marketplace with consumer and datacenter GPUs.

Sponsored placeholder links: LocalFit may earn a commission if these become affiliate links. They never influence fit results, estimates or picks.