NVIDIA Nemotron 3.5 Lightning 30B-A3B
nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16
Files and memory at 8K context
| Quant | File | Files | + KV + runtime | Smallest fit | Source |
|---|---|---|---|---|---|
| Q4_K_M bartowski/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF | 23.73 GiB | 1 | 24.8 GiB+ | 32 GB GPU | HF files |
| Q5_K_M bartowski/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF | 25.11 GiB | 1 | 26.2 GiB+ | 32 GB GPU | HF files |
| Q8_0 bartowski/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF | 32.60 GiB | 1 | 33.6 GiB+ | 48 GB GPU | HF files |
| MLX 4-bit mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-4bit | 16.55 GiB | 4 | 17.6 GiB+ | 32 GB Mac | HF files |
| MLX 8-bit mlx-community/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-8bit | 31.26 GiB | 7 | 32.3 GiB+ | 48 GB Mac | HF files |
No verified file: FP16. Sizes are exact file bytes (text weights; projectors listed separately). “+” = recurrent state or other terms unknown. Smallest fit adds 0.5 GiB spare; Macs use 75% of memory.
Quality
Artificial Analysis Intelligence Index
12.9
Variant: Nemotron 3.5 Lightning · observed 2026-10-17 · source
Rank #49 of 75 researched 2026 models.
Arena text rating
1,329
95% CI 1323–1334 · 19,938 votes · published 2026-10-15 · dataset
Exact listed model name; hosted reasoning/quantization may differ from local checkpoint.
Scores describe the evaluated hosted variant, not each quantized file. AA and Arena are never averaged.
Reported speeds 10
| Hardware | Quant | Backend | Prompt t/s | Gen t/s | Date | Source |
|---|---|---|---|---|---|---|
| M4 Max (40c GPU) 128 GB | MLX 4bit | MLX | 1,615 | 147 | 2026-10-15 | omlx.ai |
| M4 Pro (16c GPU) 64 GB | MLX 4bit | MLX | 680 | 91.5 | 2026-10-11 | omlx.ai |
| M5 Max (40c GPU) 128 GB | MLX 4bit | MLX | 1,800 | 173 | 2026-10-11 | omlx.ai |
| M2 Max (30c GPU) 64 GB | MLX 4bit | MLX | 720 | 87.1 | 2026-10-10 | omlx.ai |
| M3 Ultra (60c GPU) 96 GB | MLX 4bit | MLX | 2,049 | 136 | 2026-09-18 | omlx.ai |
| M1 Max (24c GPU) 64 GB | MLX 4bit | MLX | 359 | 61.5 | 2026-08-17 | omlx.ai |
| M3 Max (40c GPU) 48 GB | MLX 4bit | MLX | 1,326 | 116 | 2026-08-14 | omlx.ai |
| NVIDIA GeForce RTX 3090 24 GB + 64 GB system RAM | Q6_K | llama.cpp CUDA (llama-server, -ngl 999 -ncmoe 22 -np 1 -fa on --jinja) | 691 derived | 60.6 | 2026-08-11 | huggingface.co |
| NVIDIA GeForce RTX 3090 24 GB + 64 GB system RAM | Q6_K | llama.cpp CUDA (llama-server, -ngl 999 -ncmoe 22 -np 1 -fa on --jinja) | 763 derived | 73 | 2026-08-11 | huggingface.co |
| NVIDIA GeForce RTX 3090 24 GB + 64 GB system RAM | Q6_K | llama.cpp CUDA (llama-server, -ngl 999 -ncmoe 22 -np 1 -fa on --jinja) | 751 derived | 71 | 2026-08-11 | huggingface.co |
Reported by the linked sources, not reproduced by LocalFit. All rows in the speed database
Community threads
- Benchmark: fine-tuned Nemotron 3 Nano vs. nine frontier LLMs on B2B propensity ranking Hugging Face · 2026-08-24
- RTX PRO 6000 96GB Hugging Face · 2026-08-19
Qualitative signals only; titles are not verified claims.