Llama 2 7B OlderGated

Released2023-07-18Size6.7B tensorsRoleAssistantLicensellama2Contextunknown

Where it runs 8K context · ≥ 5 t/s floor

Setup	Best plan	Memory	Generate
8 GB GPU + 32 GB RAM	native F32/F16vLLMGPU streams weights from RAM · 7 GiB in RAM	KV unknown	Unknown
12 GB GPU + 32 GB RAM	native F32/F16vLLMGPU streams weights from RAM · 3 GiB in RAM	KV unknown	Unknown
16 GB GPU + 64 GB RAM	native F32/F16vLLMFull GPU	KV unknown	Unknown
24 GB GPU + 64 GB RAM	native F32/F16vLLMFull GPU	KV unknown	Unknown
32 GB GPU + 128 GB RAM	native F32/F16vLLMFull GPU	KV unknown	Unknown
2× 24 GB GPUs + 128 GB RAM	native F32/F16vLLMFull GPU	KV unknown	Unknown
96 GB workstation GPU + 256 GB RAM	native F32/F16vLLMFull GPU	KV unknown	Unknown
Mac, 24 GB	Does not fit
Mac, 48 GB	Does not fit
Mac, 128 GB	Does not fit
Mac, 512 GB	Does not fit
DGX Spark 128 GB	native F32/F16vLLMUnified memory	KV unknown	Unknown
Ryzen AI Max+ 395, 128 GB	native F32/F16vLLMUnified memory	KV unknown	Unknown
CPU only, 64 GB DDR5-5600 dual channel	Does not fit

Same plan engine as the planner, with its default reserves and RAM. How plans are chosen.

Files exact Hugging Face bytes

Artifact	Size	Files	Revision	Source
Native safetensors mixed/custom native; ; stored tensor dtypes=F32/F16 · F32/F1 meta-llama/Llama-2-7b-hf	12.55 GiB	2	01c7f73d77	HF files

Not found in the representative quant search (not proven unavailable): GGUF FP16, GGUF Q8_0, GGUF Q5_K_M, GGUF Q4_K_M, MLX 8-bit, MLX 4-bit, EXL3, AWQ, GPTQ. Complete selected file sets; projectors, MTP/draft and lookup files are listed separately and budgeted only by plans that use them. File bytes are not allocated memory.

Quality

No Artificial Analysis evaluation mapped to this checkpoint.

Arena text rating

Unknown

Publisher-reported benchmarks

None extracted from the model card.

Open LLM Leaderboard (historical): 8.81 average · dataset. Historical Open LLM Leaderboard row (dataset last modified 2025-03-20); not 2026 coverage.

AA and Arena describe the hosted model in the stated mode, not each quantized file; scores are never inherited by base, pruned, abliterated or quantized variants and never averaged together.

How to run it

No model-specific recipe in the research. The planner binds the backend’s documented command template (llama.cpp, Ollama, LM Studio, MLX, vLLM) to your plan: open a plan’s details.

Recipes are source examples (settings are the publisher’s or author’s, not measured optima) and were not executed by LocalFit. A downloadable file does not prove your backend build supports the architecture.

Reported speeds 2521

Hardware	Quant	Backend	Placement	Prompt t/s	Gen t/s	Date	Source
M1 Pro, 16 GPU cores	F16	llama.cpp Metal	Full GPU	302	13	2026-10-01	llama.cpp #4167
M1 Pro, 16 GPU cores	Q8_0	llama.cpp Metal	Full GPU	270	22	2026-10-01	llama.cpp #4167
M2 Ultra, 76 GPU cores	F16	llama.cpp Metal	Full GPU	1,402	41	2026-10-01	llama.cpp #4167
M2 Ultra, 76 GPU cores	Q8_0	llama.cpp Metal	Full GPU	1,249	67	2026-10-01	llama.cpp #4167
M2 Ultra, 76 GPU cores	F16	llama.cpp Metal	Full GPU	1,526	43	2026-10-01	llama.cpp #4167
M2 Ultra, 76 GPU cores	Q8_0	llama.cpp Metal	Full GPU	1,368	73	2026-10-01	llama.cpp #4167
M2 Ultra, 76 GPU cores	F16	llama.cpp Metal	Full GPU	1,561	43	2026-10-01	llama.cpp #4167
M2 Ultra, 76 GPU cores	Q8_0	llama.cpp Metal	Full GPU	1,387	73	2026-10-01	llama.cpp #4167
M2 Ultra, 76 GPU cores	F16	llama.cpp Metal	Full GPU	1,628	50	2026-10-01	llama.cpp #4167
M2 Ultra, 76 GPU cores	Q8_0	llama.cpp Metal	Full GPU	1,487	83	2026-10-01	llama.cpp #4167
M5 Ultra, 80 GPU cores	F16	llama.cpp Metal	Full GPU	5,207	74	2026-09-30	llama.cpp #4167
M5 Ultra, 80 GPU cores	Q8_0	llama.cpp Metal	Full GPU	4,911	119	2026-09-30	llama.cpp #4167
M5 Pro, 16 GPU cores	F16	llama.cpp Metal	Full GPU	1,311	21	2026-09-24	llama.cpp #4167
M5 Pro, 16 GPU cores	Q8_0	llama.cpp Metal	Full GPU	1,303	39	2026-09-24	llama.cpp #4167
M5 Pro, 20 GPU cores	F16	llama.cpp Metal	Full GPU	1,589	22	2026-08-31	llama.cpp #4167
M5 Pro, 20 GPU cores	Q8_0	llama.cpp Metal	Full GPU	1,554	39	2026-08-31	llama.cpp #4167
M5, 10 GPU cores	Q8_0	llama.cpp Metal	Full GPU	715	18	2026-08-30	llama.cpp #4167
M1 Ultra, 64 GPU cores	F16	llama.cpp Metal	Full GPU	1,294	48	2026-08-28	llama.cpp #4167
M1 Ultra, 64 GPU cores	Q8_0	llama.cpp Metal	Full GPU	1,168	79	2026-08-28	llama.cpp #4167
M5 Max, 40 GPU cores	F16	llama.cpp Metal	Full GPU	3,158	37	2026-08-25	llama.cpp #4167
M5 Max, 40 GPU cores	Q8_0	llama.cpp Metal	Full GPU	3,144	72	2026-08-25	llama.cpp #4167
M5 Max, 40 GPU cores (MBP 16)	F16	llama.cpp Metal	Full GPU	3,190	35	2026-06-22	llama.cpp #4167
M5 Max, 40 GPU cores (MBP 16)	Q8_0	llama.cpp Metal	Full GPU	3,220	73	2026-06-22	llama.cpp #4167
M5 Pro, 16 GPU cores (MBP 14)	F16	llama.cpp Metal	Full GPU	1,310	21	2026-04-14	llama.cpp #4167
M5 Pro, 16 GPU cores (MBP 14)	Q8_0	llama.cpp Metal	Full GPU	1,305	38	2026-04-14	llama.cpp #4167
M5 Pro, 16 GPU cores (MBP 14)	F16	llama.cpp Metal	Full GPU	696	21	2026-04-14	llama.cpp #4167
M5 Pro, 16 GPU cores (MBP 14)	Q8_0	llama.cpp Metal	Full GPU	964	38	2026-04-14	llama.cpp #4167
M5 Pro, 16 GPU cores (MBP 14)	F16	llama.cpp Metal	Full GPU	423	20	2026-04-14	llama.cpp #4167
M5 Pro, 16 GPU cores (MBP 14)	Q8_0	llama.cpp Metal	Full GPU	431	36	2026-04-14	llama.cpp #4167
M5 Max, 40 GPU cores (MBP 16)	F16	llama.cpp Metal	Full GPU	1,032	38	2026-03-31	llama.cpp #4167
M5 Max, 40 GPU cores (MBP 16)	Q8_0	llama.cpp Metal	Full GPU	1,055	65	2026-03-31	llama.cpp #4167
M5 Max, 40 GPU cores (MBP 16)	F16	llama.cpp Metal	Full GPU	1,018	38	2026-03-28	llama.cpp #4167
M5 Max, 40 GPU cores (MBP 16)	Q8_0	llama.cpp Metal	Full GPU	1,052	65	2026-03-28	llama.cpp #4167
M5, 10 GPU cores	F16	llama.cpp Metal	Full GPU	627	8.9	2026-03-28	llama.cpp #4167
M5, 10 GPU cores	Q8_0	llama.cpp Metal	Full GPU	691	16	2026-03-28	llama.cpp #4167
Mac Studio M1 Ultra (64 core GPU, 128GB)	Q4_K_M	llama.cpp Metal	Full GPU	1,004	75	2026-02-07	ai-benchmarks #48
M4 Max, 32 GPU cores	F16	llama.cpp Metal	Full GPU	736	24	2026-01-16	llama.cpp #4167
M4 Max, 32 GPU cores	Q8_0	llama.cpp Metal	Full GPU	719	44	2026-01-16	llama.cpp #4167
NVIDIA GeForce RTX 4070 Ti (Intel Core Ultra 265K custom PC)	Q4_K_M	llama.cpp Vulkan	Full GPU	5,062	104	2025-12-10	ai-benchmarks #45
NVIDIA GeForce RTX 4090 (Intel Core Ultra 265K custom PC)	Q4_K_M	llama.cpp Vulkan	Full GPU	9,653	183	2025-12-09	ai-benchmarks #43

Reported by the linked sources, not reproduced by LocalFit. Stream pieces/s and method-unknown rates are flagged and cannot pass a tokens/s floor. Showing 40 of 2521. All rows