LocalFit
Submit a benchmark

Intel Gaudi3 PCIe HL-338: what runs best?

RTX 4090RTX 5090RTX 3090RTX 4060 Ti (16 GB)RTX 5070 TiRadeon RX 7900 XTXM4 Max 128 GBM3 Ultra 512 GBDGX Spark (128 GB)All devices

Intel · Accelerator

Intel Gaudi3 PCIe HL-338

Memory128 GB HBM2eBandwidth3,700 GB/s vendor specPlussystem RAM for offload (set below)

GPUs

  • Gaudi3 PCIe HL-338128 GB · 3,700 GB/s1
Custom GPU

System RAM

≈ 89.6 GB/s theoretical ceiling (MT/s × 8 bytes × channels). DDR5’s two 32-bit subchannels count as one channel.

CPU

Used for CPU layers/experts and the -t flag. A reference platform fills its maximum channels and speed; set what you actually populated.

Context
KV cache
Mode
Advanced: reserves, bandwidth, workload, models

Defaults are editorial planning choices, not measurements: 1 GiB runtime + 0.5 GiB margin per device, 75% of unified memory, 8 GiB RAM for the OS, a vendor “24 GB” treated as 24 GiB (enter your driver’s available VRAM to override), 5 t/s minimum speed (research editorial default). Memory ledger

No GPU (CPU only)64 GB DDR5-5600 ×2 ch · 56.0 GiB usable · 89.6 GB/s theoretical8K contextF16 KVany backend≥ 5 t/s

Best plans CPU only

Best quality at ≥ 5 t/s

K2-Horizon-MoVA-36B-A4B

25.3 AA index single mode

Q4_K_Mllama.cpp CPUCPU only

~6.2–7.3 t/s Estimated extrapolated

Speed extrapolated, not measured on this setup.

Faster tradeoff

Qwen3.6-35B-A3B

18.2 AA index Reasoning

Q4_K_Mllama.cpp CPUCPU only

~11–13 t/s Estimated extrapolated

Next point on the quality/speed front; then MiniCPM5-2B.

Possibly better · speed not established

No higher-scoring model has a feasible plan here.

WeightsKV cacheReserveFreeFits* = a state term unsizedMeasuredlinks to its sourceEstimatedcalibrated or extrapolated range, ~Ceilingupper bound, ≤Unknownnever zero

Meets your 5 t/s floor 19

Best plan per model that reaches your floor by a matching measurement, a calibrated range, or an extrapolated estimate (labelled). Sorted by the selected quality metric, then speed.

  1. 1

    K2-Horizon-MoVA-36B-A4B

    IFM/K2-Horizon-MoVA-36B-A4B

    2026-09-0336B MoE · 4B active

    AA index25.3single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits
    RAM22.3 / 56.0 GiB
    Generate~6.2–7.3 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
  2. 2

    Qwen3.6-35B-A3B

    Qwen/Qwen3.6-35B-A3B

    2026-04-1635B MoE · 3B active

    AA index18.2Reasoning
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits*
    RAM19.2 / 56.0 GiB
    Generate~11–13 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
  3. 3

    K2-Horizon-3.7B

    IFM/K2-Horizon-3.7B

    2026-09-035.1B tensors · dense

    AA index15.6single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits
    RAM4.1 / 56.0 GiB
    Generate~5.4–6.4 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
  4. 4

    Qwen3.5-4B

    Qwen/Qwen3.5-4B

    2026-03-024B dense

    AA index13.1 est.Reasoning
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits*
    RAM3.1 / 56.0 GiB
    Generate~6.5–7.7 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
  5. 5

    NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

    nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16

    2026-08-1130B MoE · 3B active

    AA index12.9single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits*
    RAM23.6 / 56.0 GiB
    Generate~8.0–9.4 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
  6. 6

    MiniCPM5-2B

    openbmb/MiniCPM5-2B

    2026-09-072.5B dense

    AA index12.5single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits
    RAM1.8 / 56.0 GiB
    Generate~12–14 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
  7. 7

    Hypernova-60B-2605

    MultiverseComputingCAI/Hypernova-60B-2605

    2026-05-0660B MoE · 4.8B active

    Also: HyperNova-60B (listed with its dated update); Hypernova-60B-2602 (dated update, not separately evaluated)

    AA index11.7 est.High
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits
    RAM38.1 / 56.0 GiB
    Generate~6.0–7.1 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
  8. 8

    Nemotron-Cascade-2-30B-A3B

    nvidia/Nemotron-Cascade-2-30B-A3B

    2026-03-1930B MoE · 3B active

    AA index11.7 est.single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits*
    RAM23.1 / 56.0 GiB
    Generate~8.1–9.6 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
  9. 9

    G9v3-3B

    ai9stars/G9v3-3B

    2026-07 (day unknown)3B dense

    AA index10.8 est.single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits
    RAM2.2 / 56.0 GiB
    Generate~9.6–11 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
  10. 10

    granite-4.2-3b

    ibm-granite/granite-4.2-3b

    2026-08-253B dense

    AA index9.1single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits
    RAM2.7 / 56.0 GiB
    Generate~7.9–9.3 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
  11. 11

    LFM2.5-2.6B

    LiquidAI/LFM2.5-2.6B

    2026-08-042.7B dense

    AA index8.4 est.single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits*
    RAM1.7 / 56.0 GiB
    Generate~12–14 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
  12. 12

    LFM2.5-8B-A1B

    LiquidAI/LFM2.5-8B-A1B

    2026-05-288.3B MoE · 1.5B active

    AA index7.2 est.single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits*
    RAM4.9 / 56.0 GiB
    Generate~21–25 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
  13. 13

    Qwen3.5-2B

    Qwen/Qwen3.5-2B

    2026-03-022B dense

    AA index6.9 est.Reasoning
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits*
    RAM1.4 / 56.0 GiB
    Generate~14–17 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
  14. 14

    Qwen3.5-0.8B

    Qwen/Qwen3.5-0.8B

    2026-03-020.8B dense

    AA index6.1 est.Reasoning
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits*
    RAM0.6 / 56.0 GiB
    Generate~32–38 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
  15. 15

    LFM2.5-1.2B-ThinkingSpecialist

    LiquidAI/LFM2.5-1.2B-Thinking

    2026-01-201.2B tensors · dense

    AA index5.2 est.single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits*
    RAM0.8 / 56.0 GiB
    Generate~26–31 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
  16. 16

    LFM2.5-1.2B-Instruct

    LiquidAI/LFM2.5-1.2B-Instruct

    2026-01-051.2B tensors · dense

    AA index5.2 est.single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits*
    RAM0.8 / 56.0 GiB
    Generate~26–31 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
  17. 17

    LFM2.5-VL-1.6BSpecialist

    LiquidAI/LFM2.5-VL-1.6B

    2026-01-051.6B tensors · dense

    AA index4.8 est.single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits*
    RAM0.8 / 56.0 GiB
    Generate~26–31 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
  18. 18

    NVIDIA-Nemotron-3-Nano-4B-BF16

    nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16

    2026-03-164B dense

    AA index4.8single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits*
    RAM2.8 / 56.0 GiB
    Generate~7.0–8.3 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
  19. 19

    K2-Horizon-0.9B

    IFM/K2-Horizon-0.9B

    2026-09-031.1B tensors · dense

    AA index3.0single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits
    RAM1.1 / 56.0 GiB
    Generate~23–27 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
+ 23 models without a AA index score

Memory not fully resolved 10

The weights fit, but the KV cache for this attention type (or the chosen cache layout) is not established, so a fit cannot be guaranteed.

  1. 1

    gemma-4-26B-A4B-it

    google/gemma-4-26B-A4B-it

    2026-04-0225B MoE · 3.8B active

    AA index16.7 est.Reasoning
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyKV unknown
    RAM15.9 / 56.0 GiB
    Generate~7.9–9.4 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
  2. 2

    LongCat-Flash-Lite

    meituan-longcat/LongCat-Flash-Lite

    2026-01 (day unknown)69B MoE · 3B active

    AA index11.5 est.single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyKV unknown
    RAM37.5 / 56.0 GiB
    Generate~12–14 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
  3. 3

    Ling-3.0-tiny

    inclusionAI/Ling-3.0-tiny

    2026-08 (day unknown)7.9B MoE · 1.3B active

    AA index11.1single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyKV unknown
    RAM4.6 / 56.0 GiB
    Generate~25–30 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
  4. 4

    Qwen3-Next-80B-A3B-InstructOlder

    Qwen/Qwen3-Next-80B-A3B-Instruct

    2025-09-1181B tensors · MoE

    AA index9.6 est.single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyKV unknown
    RAM45.2 / 56.0 GiB
    Generate~6.2–7.3 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
  5. 5

    diffusiongemma-26B-A4B-it

    google/diffusiongemma-26B-A4B-it

    2026-06-1025B MoE · 3.8B active

    AA index9.5 est.single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyKV unknown
    RAM15.7 / 56.0 GiB
    Generate~8.0–9.5 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
  6. 6

    Qwen3-Coder-NextSpecialist

    Qwen/Qwen3-Coder-Next

    2026-02-0280B MoE · 3B active

    AA index9.2single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyKV unknown
    RAM45.4 / 56.0 GiB
    Generate~11–13 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
  7. 7

    gemma-4-E2B-it

    google/gemma-4-E2B-it

    2026-04-025.1B dense

    AA index7.8 est.Reasoning
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyKV unknown
    RAM3.2 / 56.0 GiB
    Generate~5.9–7.0 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
  8. 8

    Llama 3.3 70B InstructOlderGated

    meta-llama/Llama-3.3-70B-Instruct

    2024-12-0671B tensors

    AA index7.7 est.single mode
    Plan & memory
    Q5_K_Mllama.cpp CPUCPU onlyKV unknown
    RAM46.5 / 56.0 GiB
    Generate—Unknownspeed unknown
    First token · 4K prompt—Unknown
  9. 9

    Phi-4-mini-instructOlder

    microsoft/Phi-4-mini-instruct

    2025-02-263.8B tensors · dense

    AA index6.3 est.single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyKV unknown
    RAM2.3 / 56.0 GiB
    Generate~8.2–9.7 t/sEstimatedextrapolated≈ meets 5 t/s
    First token · 4K prompt—Unknown
  10. 10

    tiny-aya-globalGated

    CohereLabs/tiny-aya-global

    2026-02-173.3B tensors

    AA index4.8 est.single mode
    Plan & memory
    FP16llama.cpp CPUCPU onlyKV unknown
    RAM6.2 / 56.0 GiB
    Generate—Unknownspeed unknown
    First token · 4K prompt—Unknown
+ 15 models without a AA index score

Speed unknown 5

No measurement, calibration or bandwidth figure for this plan.

  1. 1

    gpt-oss-20bOlder

    openai/gpt-oss-20b

    2025-08-0521B tensors · MoE

    AA index9.0High
    Plan & memory
    Q8_0llama.cpp CPUCPU onlyFits
    RAM11.5 / 56.0 GiB
    Generate—Unknownspeed unknown
    First token · 4K prompt—Unknown
  2. 2

    granite-4.0-h-1b

    ibm-granite/granite-4.0-h-1b

    2025-10-281.5B tensors · MoE

    AA index5.2 est.single mode
    Plan & memory
    Q8_0llama.cpp CPUCPU onlyFits*
    RAM1.5 / 56.0 GiB
    Generate—Unknownspeed unknown
    First token · 4K prompt—Unknown
  3. 3

    granite-4.0-micro

    ibm-granite/granite-4.0-micro

    2025-10-023.4B tensors · MoE

    AA index5.1 est.single mode
    Plan & memory
    Q8_0llama.cpp CPUCPU onlyFits
    RAM4.0 / 56.0 GiB
    Generate—Unknownspeed unknown
    First token · 4K prompt—Unknown
  4. 4

    granite-4.0-350m

    ibm-granite/granite-4.0-350m

    2025-10-280.4B tensors · MoE

    AA index4.8 est.single mode
    Plan & memory
    Q8_0llama.cpp CPUCPU onlyFits
    RAM0.6 / 56.0 GiB
    Generate—Unknownspeed unknown
    First token · 4K prompt—Unknown
  5. 5

    granite-4.0-h-350m

    ibm-granite/granite-4.0-h-350m

    2025-10-280.3B tensors · MoE

    AA index4.8 est.single mode
    Plan & memory
    Q8_0llama.cpp CPUCPU onlyFits*
    RAM0.4 / 56.0 GiB
    Generate—Unknownspeed unknown
    First token · 4K prompt—Unknown
+ 4 models without a AA index score

Below 5 t/s 20

Fits, but the best evidence (measured, estimated or the bandwidth ceiling) is under your floor. Lower the floor or add memory bandwidth.

  1. 1

    Qwen3.8-27B

    Qwen/Qwen3.8-27B

    2026-08-1427B dense

    AA index33.7Xhigh
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits*
    RAM18.2 / 56.0 GiB
    Generate~1.1–1.3 t/sEstimatedextrapolatedbelow 5 t/s
    First token · 4K prompt—Unknown
  2. 2

    K2-Horizon-7B

    IFM/K2-Horizon-7B

    2026-09-039B tensors · dense

    AA index20.6single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits
    RAM6.3 / 56.0 GiB
    Generate~3.3–3.9 t/sEstimatedextrapolatedbelow 5 t/s
    First token · 4K prompt—Unknown
  3. 3

    Muse-Glimmer-30B

    meta-models/Muse-Glimmer-30B

    2026-08-1030B dense

    AA index17.5High
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits
    RAM15.8 / 56.0 GiB
    Generate~1.2–1.4 t/sEstimatedextrapolatedbelow 5 t/s
    First token · 4K prompt—Unknown
  4. 4

    gemma-4-31B-it

    google/gemma-4-31B-it

    2026-04-0231B dense

    AA index14.7Reasoning
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyKV unknown
    RAM18.3 / 56.0 GiB
    Generate~1.0–1.2 t/sEstimatedextrapolatedbelow 5 t/s
    First token · 4K prompt—Unknown
  5. 5

    gemma-4-12B-it

    google/gemma-4-12B-it

    2026-06-0312B dense

    AA index14.2 est.Reasoning
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyKV unknown
    RAM7.1 / 56.0 GiB
    Generate~2.7–3.1 t/sEstimatedextrapolatedbelow 5 t/s
    First token · 4K prompt—Unknown
  6. 6

    EXAONE-4.5-33B

    LGAI-EXAONE/EXAONE-4.5-33B

    2026-04-0933B dense

    AA index13.2 est.Reasoning
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits
    RAM19.9 / 56.0 GiB
    Generate~1.0–1.1 t/sEstimatedextrapolatedbelow 5 t/s
    First token · 4K prompt—Unknown
  7. 7

    granite-4.2-30b

    ibm-granite/granite-4.2-30b

    2026-08-2530B dense

    AA index12.7single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits
    RAM18.5 / 56.0 GiB
    Generate~1.1–1.3 t/sEstimatedextrapolatedbelow 5 t/s
    First token · 4K prompt—Unknown
  8. 8

    Qwen3.5-9B

    Qwen/Qwen3.5-9B

    2026-03-029B dense

    AA index11.2Reasoning
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits*
    RAM6.0 / 56.0 GiB
    Generate~3.2–3.8 t/sEstimatedextrapolatedbelow 5 t/s
    First token · 4K prompt—Unknown
  9. 9

    granite-4.2-8b

    ibm-granite/granite-4.2-8b

    2026-08-258B dense

    AA index11.1single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits
    RAM6.2 / 56.0 GiB
    Generate~3.4–4.0 t/sEstimatedextrapolatedbelow 5 t/s
    First token · 4K prompt—Unknown
  10. 10

    Tri-21B-Think

    trillionlabs/Tri-21B-Think

    2026-02 (day unknown)21B dense

    AA index9.0 est.single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits
    RAM13.3 / 56.0 GiB
    Generate~1.5–1.8 t/sEstimatedextrapolatedbelow 5 t/s
    First token · 4K prompt—Unknown
  11. 11

    gemma-4-E4B-it

    google/gemma-4-E4B-it

    2026-04-028B dense

    AA index8.9Reasoning
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyKV unknown
    RAM5.0 / 56.0 GiB
    Generate~3.8–4.5 t/sEstimatedextrapolatedbelow 5 t/s
    First token · 4K prompt—Unknown
  12. 12

    Olmo-3.1-32B-ThinkSpecialist

    allenai/Olmo-3.1-32B-Think

    2025-12-1232B tensors · dense

    AA index7.1 est.single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits
    RAM19.4 / 56.0 GiB
    Generate~1.0–1.2 t/sEstimatedextrapolatedbelow 5 t/s
    First token · 4K prompt—Unknown
  13. 13

    Olmo-3.1-32B-Instruct

    allenai/Olmo-3.1-32B-Instruct

    2025-12-1232B tensors · dense

    AA index6.5 est.single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits
    RAM19.4 / 56.0 GiB
    Generate~1.0–1.2 t/sEstimatedextrapolatedbelow 5 t/s
    First token · 4K prompt—Unknown
  14. 14

    granite-4.0-h-small

    ibm-granite/granite-4.0-h-small

    2025-10-0232B tensors · MoE

    AA index6.0 est.single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits*
    RAM18.3 / 56.0 GiB
    Generate~3.8–4.5 t/sEstimatedextrapolatedbelow 5 t/s
    First token · 4K prompt—Unknown
  15. 15

    Phi-4Older

    microsoft/phi-4

    2024-12-1215B tensors · dense

    AA index5.9 est.single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits
    RAM9.8 / 56.0 GiB
    Generate~2.1–2.5 t/sEstimatedextrapolatedbelow 5 t/s
    First token · 4K prompt—Unknown
  16. 16

    Olmo-3-7B-ThinkSpecialist

    allenai/Olmo-3-7B-Think

    2025-11-207.3B tensors · dense

    AA index5.6 est.single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits
    RAM6.7 / 56.0 GiB
    Generate~3.1–3.6 t/sEstimatedextrapolatedbelow 5 t/s
    First token · 4K prompt—Unknown
  17. 17

    Olmo-3-7B-Instruct

    allenai/Olmo-3-7B-Instruct

    2025-11-207.3B tensors · dense

    AA index5.2 est.single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits
    RAM6.7 / 56.0 GiB
    Generate~3.1–3.6 t/sEstimatedextrapolatedbelow 5 t/s
    First token · 4K prompt—Unknown
  18. 18

    Apertus-70B-Instruct-2509

    swiss-ai/Apertus-70B-Instruct-2509

    2025-09-0271B tensors · dense

    AA index5.1 est.single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits
    RAM43.2 / 56.0 GiB
    Generate~0.5–0.5 t/sEstimatedextrapolatedbelow 5 t/s
    First token · 4K prompt—Unknown
  19. 19

    Molmo2-8BSpecialist

    allenai/Molmo2-8B

    2025-12-118.7B tensors · dense

    AA index5.0 est.single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits
    RAM5.8 / 56.0 GiB
    Generate~3.6–4.3 t/sEstimatedextrapolatedbelow 5 t/s
    First token · 4K prompt—Unknown
  20. 20

    Apertus-8B-Instruct-2509

    swiss-ai/Apertus-8B-Instruct-2509

    2025-09-028.1B tensors · dense

    AA index4.8 est.single mode
    Plan & memory
    Q4_K_Mllama.cpp CPUCPU onlyFits
    RAM5.7 / 56.0 GiB
    Generate~3.6–4.3 t/sEstimatedextrapolatedbelow 5 t/s
    First token · 4K prompt—Unknown
+ 44 models without a AA index score
97 models with no feasible plan on this setup

Specs & sources

Memory128 GB HBM2esource
Bandwidth3,700 GB/ssource
FP16unknownNo unambiguous comparable value verified; never substitute AI TOPS, BF16, or whole-system power.
Power600 Wsource

Doesn't fit? Rent a GPU

  • RunpodPlaceholderCloud GPU pods you can rent for running larger models.
  • Vast.aiPlaceholderGPU rental marketplace with many hosts and GPU types.
  • LambdaPlaceholderCloud GPU instances for AI training and inference.
  • TensorDockPlaceholderGPU cloud marketplace with consumer and datacenter GPUs.

Sponsored placeholder links: LocalFit may earn a commission if these become affiliate links. They never influence fit results, estimates or picks.