{"schema":"localfit-site-2","generated_at":"2026-10-17","source":"research/run_recipes.json","status":"Source-backed run guidance, not independently executed or universal optimal configurations","policy":["Never execute a source command merely because it is in the research catalog. Bind exact model/version/hardware and inspect prerequisites.","Best means documented candidate configuration for a specific task and system; no universal quant/backend winner is asserted.","Publisher examples with large contexts or concurrency are not consumer-memory-fit promises.","CLI flags belong to the cited backend build; fork-only flags must never be sent to stock binaries.","Missing per-model deployment evidence remains unknown, with model-card run_sources retained in models_2026.json."],"recipes":[{"id":"dory-vllm-pr60149","kind":"model","backend":"vLLM","repo":"nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16","repos":["nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16"],"models":["nvidia-nvidia-nemotron-labs-dory1-0-4b-bf16"],"url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16/blob/b366b87f762ea334ba3eb55912031baba1515810/README.md","status":"publisher_documented_not_executed_by_researcher","hardware":"NVIDIA CUDA; exact device/memory unknown","steps":[{"code":"VLLM_USE_PRECOMPILED=1 pip install \"git+https://github.com/vllm-project/vllm.git@refs/pull/60149/head\"","url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16/blob/b366b87f762ea334ba3eb55912031baba1515810/README.md"},{"code":"vllm serve --model nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 \\\n    --hf-overrides '{\"n_recurrent_loops\": 2, \"recurrent_kv_cache_mode\": \"per_loop\"}' \\\n    --max-model-len 262144 \\\n    --max-num-seqs 128 \\\n    --enable-prefix-caching \\\n    --async-scheduling \\\n    --reasoning-parser nemotron_v3 \\\n    --tool-call-parser qwen3_coder \\\n    --enable-auto-tool-choice","url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16/blob/b366b87f762ea334ba3eb55912031baba1515810/README.md"}],"sampling":{"temperature":1,"top_p":0.95},"prerequisites":"See source setup/dependencies; these snippets were inspected, not executed.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["Support under review in PR60149; not a verified stable-release recipe.","Two loops reuse weights but per_loop caches grow with loop count. last_loop is a distinct cache mode.","Card 256K context/max_num_seqs128 is an example, not a consumer memory-fit guarantee."],"notes":[]},{"id":"dory-llamacpp-metal-pinned","kind":"model","backend":"llama.cpp","repo":"nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16","repos":["nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16"],"models":["nvidia-nvidia-nemotron-labs-dory1-0-4b-bf16"],"url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16/blob/b366b87f762ea334ba3eb55912031baba1515810/README.md","status":"publisher_documented_not_executed_by_researcher","hardware":"Apple Silicon ARM64 Metal; macOS26 SDK specified by publisher","steps":[{"code":"brew install cmake openssl@3\n\ngit init llama.cpp-dory\ncd llama.cpp-dory\ngit remote add origin https://github.com/brianchmielNV/llama.cpp.git\ngit fetch --depth 1 origin 98a74c97ea09ac72f6ceda1559808fc0a2f447ce\ngit checkout --detach FETCH_HEAD\n\nHF_UI_VERSION=b11551 cmake -S . -B build \\\n    -DCMAKE_BUILD_TYPE=Release \\\n    -DCMAKE_OSX_ARCHITECTURES=arm64 \\\n    -DCMAKE_OSX_SYSROOT=\"/Library/Developer/CommandLineTools/SDKs/MacOSX26.sdk\" \\\n    -DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON \\\n    -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=Apple \\\n    -DLLAMA_BUILD_SERVER=ON \\\n    -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=OFF -DLLAMA_BUILD_APP=OFF \\\n    -DLLAMA_BUILD_UI=OFF -DLLAMA_USE_PREBUILT_UI=ON \\\n    -DLLAMA_OPENSSL=ON -DOPENSSL_ROOT_DIR=\"$(brew --prefix openssl@3)\"\ncmake --build build --parallel 4 --target llama-server","url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16/blob/b366b87f762ea334ba3eb55912031baba1515810/README.md"},{"code":"Download dory-q5_k_m.gguf from https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-GGUF and place it in llama.cpp-dory/.","url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16/blob/b366b87f762ea334ba3eb55912031baba1515810/README.md"},{"code":"echo \"96080be49ffd67b916018b7bdef8c484762576fb52780e10ab98ac4978a660e0  dory-q5_k_m.gguf\" | shasum -a 256 -c -","url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16/blob/b366b87f762ea334ba3eb55912031baba1515810/README.md"},{"code":"./build/bin/llama-server \\\n    -m ./dory-q5_k_m.gguf \\\n    --alias dory \\\n    --host 127.0.0.1 --port 8080 \\\n    -ngl 999 \\\n    --ctx-size 4096 --parallel 1 \\\n    --batch-size 512 --ubatch-size 512 \\\n    --cache-type-k f16 --cache-type-v f16 \\\n    --flash-attn off \\\n    --no-context-shift \\\n    --jinja --reasoning on \\\n    --temp 1.0 --top-p 0.95 --top-k 0 --min-p 0 --repeat-penalty 1 \\\n    --override-kv \"dory.num_loops=int:2,dory.recurrent_kv_cache_mode=str:per_loop\"","url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16/blob/b366b87f762ea334ba3eb55912031baba1515810/README.md"}],"sampling":{"temperature":1,"top_p":0.95,"top_k":0,"min_p":0,"repeat_penalty":1},"prerequisites":"See source setup/dependencies; these snippets were inspected, not executed.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["Pinned fork 98a74c97ea09ac72f6ceda1559808fc0a2f447ce; upstream support PR30306.","Card command uses Q5_K_M, 4096 context, batch 512, FP16 KV, flash attention off, per_loop. Do not silently assume FA or quantized KV support.","Changing recurrent_kv_cache_mode to last_loop saves recurrent cache but changes execution configuration."],"notes":[]},{"id":"lightning-bf16-h100","kind":"model","backend":"vLLM","repo":"nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16","repos":["nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16"],"models":["nemotron-3.5-lightning-30b-a3b"],"url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/blob/a9904d24bcc1d289a1950fa9d2b978c47cf903b9/README.md","status":"publisher_documented_not_executed_by_researcher","hardware":"1xH10080GB, BF16","steps":[{"code":"export MODEL_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16","url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/blob/a9904d24bcc1d289a1950fa9d2b978c47cf903b9/README.md"},{"code":"vllm serve --model $MODEL_CKPT \\\n    --max-num-seqs 128 \\\n    --enable-prefix-caching \\\n    --async-scheduling \\\n    --mamba-backend flashinfer \\\n    --mamba-ssm-cache-dtype float16 \\\n    --enable-mamba-cache-stochastic-rounding \\\n    --mamba-cache-philox-rounds 5 \\\n    --reasoning-parser nemotron_v3 \\\n    --tool-call-parser qwen3_coder \\\n    --enable-auto-tool-choice","url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16/blob/a9904d24bcc1d289a1950fa9d2b978c47cf903b9/README.md"}],"sampling":{"temperature":1,"top_p":0.95},"prerequisites":"Publisher image vllm/vllm-openai: v 0.27.1.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["Publisher validates 256K on single 80GB H100; longer 1M recipe is separate.","Mamba SSM cache dtype/stochastic rounding and Philox rounds are part of numerical behavior.","Lower max_model_len/concurrency to match memory; no consumer GPU fit inherited.","Source H100 section states 256K but command omits max-model-len. Explicitly constrain context to 262144 when adapting it; no successful local test was performed."],"notes":[]},{"id":"lightning-local-gguf","kind":"model","backend":"llama.cpp","repo":"nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16","repos":["nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16"],"models":["nemotron-3.5-lightning-30b-a3b"],"url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4/raw/main/README.md","status":"publisher_or_backend_documented_not_executed_by_researcher","hardware":"Publisher section: RTX5090,DGX Spark,RTX6000Pro","steps":[{"code":"llama-server \\\n  -hf ggml-org/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF:Q4_K_M \\\n  --temp 1.0 --top-p 0.95 \\\n  -np 1 \\\n  -c 40960 \\\n  --port 8000 \\\n  -ngl 99 \\\n  -fa on \\\n  --jinja \\\n  --no-webui \\\n  --fit off","url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4/raw/main/README.md"}],"prerequisites":"See linked source for backend installation/version requirements.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["Exact Q4_K_M artifact family and 40,960 context in publisher command.","Quantized weight support does not establish full resident fit or throughput.","NVFP4 tensor-core path has separate limitations; this is GGUF."],"notes":[]},{"id":"lightning-spark-nvfp4-dspark","kind":"model","backend":"vLLM","repo":"nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16","repos":["nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16"],"models":["nemotron-3.5-lightning-30b-a3b"],"url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4/raw/main/README.md","status":"publisher_or_backend_documented_not_executed_by_researcher","hardware":"DGX SparkGB10","steps":[{"code":"export MODEL_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4\nexport DSPARK_CKPT=nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4-DSpark","url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4/raw/main/README.md"},{"code":"vllm serve --model $MODEL_CKPT \\\n  --moe-backend marlin \\\n  --kv-cache-dtype fp8 \\\n  --enable-prefix-caching \\\n  --gpu-memory-utilization 0.85 \\\n  --speculative_config.num_speculative_tokens 3 \\\n  --mamba-backend flashinfer \\\n  --mamba-cache-mode align \\\n  --reasoning-parser nemotron_v3 \\\n  --speculative_config.model $DSPARK_CKPT \\\n  --tool-call-parser qwen3_coder \\\n  --enable-auto-tool-choice","url":"https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4/raw/main/README.md"}],"prerequisites":"vllm/vllm-openai: v 0.27.1; model and DSpark checkpoint environment variables from source.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["Marlin W4A16 on GB10, not native FP4 tensor-core inference.","Separate DSpark draft model adds weights/cache and is recommended for low-concurrency workloads; maximum throughput H100 recipe disables speculation."],"notes":[]},{"id":"qwen38-27b-single5090","kind":"model","backend":"vLLM","repo":"Qwen/Qwen3.8-27B","repos":["Qwen/Qwen3.8-27B"],"models":["qwen3.8-27b"],"url":"https://recipes.vllm.ai/Qwen/Qwen3.8-27B","status":"publisher_or_backend_documented_not_executed_by_researcher","hardware":"1xRTX5090","steps":[{"code":"vllm serve Inferact/Qwen3.8-27B-NVFP4 \\\n  --tensor-parallel-size 1 \\\n  --max-model-len 32768 \\\n  --kv-cache-dtype fp8 \\\n  --enforce-eager \\\n  --reasoning-parser qwen3","url":"https://recipes.vllm.ai/Qwen/Qwen3.8-27B"}],"prerequisites":"Backend recipe header lists vLLM>=0.17.0; specific RTX5090 validation uses 0.26.1rc1.dev608+g99a10304d. Transformers>=5.8.0 for processor. Do not infer exact consumer support from generic minimum alone.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["Source-specific Inferact NVFP4 build; unsloth NVFP4 is not memory-interchangeable on 32GB cards.","Single RTX5090 requires enforce_eager in documented 32K plan; CUDA graphs consume additional memory.","FP8 KV in this vLLM plan does not imply FP8/quantized recurrent-state support in other backends."],"notes":[]},{"id":"qwen38-27b-two5090","kind":"model","backend":"vLLM","repo":"Qwen/Qwen3.8-27B","repos":["Qwen/Qwen3.8-27B"],"models":["qwen3.8-27b"],"url":"https://recipes.vllm.ai/Qwen/Qwen3.8-27B","status":"publisher_or_backend_documented_not_executed_by_researcher","hardware":"2xRTX5090","steps":[{"code":"vllm serve unsloth/Qwen3.8-27B-NVFP4 \\\n  --tensor-parallel-size 2 \\\n  --max-model-len 262144 \\\n  --kv-cache-dtype fp8 \\\n  --reasoning-parser qwen3 \\\n  --enable-auto-tool-choice --tool-call-parser qwen3_coder","url":"https://recipes.vllm.ai/Qwen/Qwen3.8-27B"}],"prerequisites":"Backend recipe header lists vLLM>=0.17.0; specific RTX5090 validation uses 0.26.1rc1.dev608+g99a10304d. Transformers>=5.8.0 for processor. Do not infer exact consumer support from generic minimum alone.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["Exact unsloth NVFP4 build and TP2; card-specific usable VRAM and graph overhead cannot be transferred to 1 GPU.","262144 context is source configuration, not arbitrary concurrency fit."],"notes":[]},{"id":"qwen35-4b-vllm-tools","kind":"model","backend":"vLLM","repo":"Qwen/Qwen3.5-4B","repos":["Qwen/Qwen3.5-4B"],"models":["qwen3.5-4b"],"url":"https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md","status":"publisher_documented_not_executed_by_researcher","hardware":"NVIDIA CUDA exact device unknown","steps":[{"code":"uv pip install vllm --torch-backend=auto --extra-index-url https://wheels.vllm.ai/nightly","url":"https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md"},{"code":"vllm serve Qwen/Qwen3.5-4B --port 8000 --tensor-parallel-size 1 --max-model-len 262144 --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder","url":"https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md"}],"sampling":{"thinking":{"temperature":1,"top_p":0.95,"top_k":20},"nonthinking":{"temperature":0.7,"top_p":0.8,"top_k":20}},"prerequisites":"See source setup/dependencies; these snippets were inspected, not executed.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["Source requests nightly vLLM; no fixed stable version established by this card.","262K context is capability ceiling, not memory budget.","language-model-only flag is a separate text-only path; vision tower/projector budgets required for images."],"notes":[]},{"id":"qwen35-4b-sglang-mtp","kind":"model","backend":"SGLang","repo":"Qwen/Qwen3.5-4B","repos":["Qwen/Qwen3.5-4B"],"models":["qwen3.5-4b"],"url":"https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md","status":"publisher_documented_not_executed_by_researcher","hardware":"unknown","steps":[{"code":"uv pip install 'git+https://github.com/sgl-project/sglang.git#subdirectory=python&egg=sglang[all]'","url":"https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md"},{"code":"python -m sglang.launch_server --model-path Qwen/Qwen3.5-4B --port 8000 --tp-size 1 --mem-fraction-static 0.8 --context-length 262144 --reasoning-parser qwen3 --speculative-algo NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4","url":"https://huggingface.co/Qwen/Qwen3.5-4B/blob/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a/README.md"}],"prerequisites":"See source setup/dependencies; these snippets were inspected, not executed.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["MTP settings are model/backend-specific. Additional speculative state/workspace must be included.","Speculative acceptance and speed are workload-dependent, not a guaranteed multiplier."],"notes":[]},{"id":"gemma4-e2b-transformers-text","kind":"model","backend":"Transformers","repo":"google/gemma-4-E2B-it","repos":["google/gemma-4-E2B-it"],"models":["gemma-4-e2b"],"url":"https://huggingface.co/google/gemma-4-E2B-it/blob/3e22461f65e89153144f8adb70e3b8c2cc9845a7/README.md","status":"publisher_documented_not_executed_by_researcher","hardware":"unknown","steps":[{"code":"from transformers import AutoProcessor, AutoModelForMultimodalLM\n\nMODEL_ID = \"google/gemma-4-E2B-it\"\n\n# Load model\nprocessor = AutoProcessor.from_pretrained(MODEL_ID)\nmodel = AutoModelForMultimodalLM.from_pretrained(\n    MODEL_ID,\n    dtype=\"auto\",\n    device_map=\"auto\"\n)","url":"https://huggingface.co/google/gemma-4-E2B-it/blob/3e22461f65e89153144f8adb70e3b8c2cc9845a7/README.md"},{"code":"# Prompt\nmessages = [\n    {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n    {\"role\": \"user\", \"content\": \"Write a short joke about saving RAM.\"},\n]\n\n# Process input\ninputs = processor.apply_chat_template(\n    messages,\n    tokenize=True,\n    return_dict=True,\n    return_tensors=\"pt\",\n    add_generation_prompt=True,\n    enable_thinking=False\n).to(model.device)\ninput_len = inputs[\"input_ids\"].shape[-1]\n\n# Generate output\noutputs = model.generate(**inputs, max_new_tokens=1024)\nresponse = processor.decode(outputs[0][input_len:], skip_special_tokens=False)\n\n# Parse output\nprocessor.parse_response(response, prefix=inputs[\"input_ids\"])","url":"https://huggingface.co/google/gemma-4-E2B-it/blob/3e22461f65e89153144f8adb70e3b8c2cc9845a7/README.md"}],"sampling":{"temperature":1,"top_p":0.95,"top_k":64},"prerequisites":"pip install -U transformers torch accelerate; no minimum version fixed by card.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["Multimodal loader is required even for text-only example.","Gemma4 layer sharing/per-layer input state/sliding attention need architecture-aware cache and runtime support.","Publisher sampling differs from Qwen."],"notes":[]},{"id":"diffusiongemma-transformers","kind":"model","backend":"Transformers block-diffusion","repo":"google/diffusiongemma-26B-A4B-it","repos":["google/diffusiongemma-26B-A4B-it"],"models":["diffusiongemma-26b-a4b"],"url":"https://huggingface.co/google/diffusiongemma-26B-A4B-it/blob/f7f5b7f5fa82ffc52addd066915886d497f5517b/README.md","status":"publisher_documented_not_executed_by_researcher","hardware":"unknown","steps":[{"code":"from transformers import DiffusionGemmaForBlockDiffusion, AutoProcessor\n\nMODEL_ID = \"google/diffusiongemma-26B-A4B-it\"\n\n# Load model\nprocessor = AutoProcessor.from_pretrained(MODEL_ID)\nmodel = DiffusionGemmaForBlockDiffusion.from_pretrained(\n    MODEL_ID,\n    dtype=\"auto\",\n    device_map=\"auto\",\n)","url":"https://huggingface.co/google/diffusiongemma-26B-A4B-it/blob/f7f5b7f5fa82ffc52addd066915886d497f5517b/README.md"},{"code":"# Prompt\nmessage = [\n    {\"role\": \"user\", \"content\": \"Why is the sky blue?\"}\n]\n\n# Process input\ninput_ids = processor.apply_chat_template(\n    message,\n    tokenize=True,\n    add_generation_prompt=True,\n    return_dict=True,\n    return_tensors=\"pt\"\n).to(model.device)\noutput = model.generate(**input_ids, max_new_tokens=512)\n\n# Parse output\ntext = processor.decode(output[0], skip_special_tokens=False)","url":"https://huggingface.co/google/diffusiongemma-26B-A4B-it/blob/f7f5b7f5fa82ffc52addd066915886d497f5517b/README.md"}],"prerequisites":"See source setup/dependencies; these snippets were inspected, not executed.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["Requires DiffusionGemmaForBlockDiffusion; ordinary autoregressive serving/latency/KV assumptions are unsupported.","Do not transfer Gemma 26 A4B autoregressive throughput to diffusion decode configuration."],"notes":[]},{"id":"lfm25-26-transformers","kind":"model","backend":"Transformers","repo":"LiquidAI/LFM2.5-2.6B","repos":["LiquidAI/LFM2.5-2.6B"],"models":["lfm2.5-2.6b"],"url":"https://huggingface.co/LiquidAI/LFM2.5-2.6B/blob/654f9463ce32b05d0429d76fe1f580b27d4c1ac0/README.md","status":"publisher_documented_not_executed_by_researcher","hardware":"unknown","steps":[{"code":"from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer\n\nmodel_id = \"LiquidAI/LFM2.5-2.6B\"\nmodel = AutoModelForCausalLM.from_pretrained(\n    model_id,\n    device_map=\"auto\",\n    dtype=\"bfloat16\",\n#   attn_implementation=\"flash_attention_2\" <- uncomment on compatible GPU\n)\ntokenizer = AutoTokenizer.from_pretrained(model_id)\nstreamer = TextStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)\n\nprompt = \"What is C. elegans?\"\n\ninput_ids = tokenizer.apply_chat_template(\n    [{\"role\": \"user\", \"content\": prompt}],\n    add_generation_prompt=True,\n    return_tensors=\"pt\",\n    tokenize=True,\n)[\"input_ids\"].to(model.device)\n\noutput = model.generate(\n    input_ids,\n    do_sample=True,\n    temperature=0.1,\n    top_k=50,\n    repetition_penalty=1.1,\n    max_new_tokens=512,\n    streamer=streamer,\n)","url":"https://huggingface.co/LiquidAI/LFM2.5-2.6B/blob/654f9463ce32b05d0429d76fe1f580b27d4c1ac0/README.md"}],"sampling":{"temperature":0.1,"top_k":50},"prerequisites":"transformers>=5.0.0.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["Publisher recommends agentic workflows, tools, extraction and RAG; explicitly not agentic coding or knowledge-heavy tasks.","Hybrid conv/state memory plus attention KV must be accounted."],"notes":[]},{"id":"lfm25-vl3-transformers","kind":"model","backend":"Transformers","repo":"LiquidAI/LFM2.5-VL-3B","repos":["LiquidAI/LFM2.5-VL-3B"],"models":["liquidai-lfm2-5-vl-3b"],"url":"https://huggingface.co/LiquidAI/LFM2.5-VL-3B/blob/35a118d938ce6d123ac2d371649f24a8efb69058/README.md","status":"publisher_documented_not_executed_by_researcher","hardware":"unknown","steps":[{"code":"import torch\nfrom transformers import AutoModelForImageTextToText, AutoProcessor\n\nmodel_id = \"LiquidAI/LFM2.5-VL-3B\"\n\nmodel = AutoModelForImageTextToText.from_pretrained(model_id, dtype=torch.bfloat16)\nprocessor = AutoProcessor.from_pretrained(model_id)\n\nmessages = [\n    {\n        \"role\": \"user\",\n        \"content\": [\n            {\"type\": \"image\", \"url\": \"https://placecats.com/300/200\"},\n            {\"type\": \"text\", \"text\": \"Describe this image.\"},\n        ],\n    }\n]\n\ninputs = processor.apply_chat_template(\n    messages,\n    add_generation_prompt=True,\n    tokenize=True,\n    return_dict=True,\n    return_tensors=\"pt\",\n).to(model.device)\n\nwith torch.inference_mode():\n    output_ids = model.generate(\n        **inputs,\n        do_sample=True,\n        temperature=0.2,\n        top_k=50,\n        repetition_penalty=1.0,\n        max_new_tokens=256,\n    )\n\ngenerated_ids = output_ids[:, inputs[\"input_ids\"].shape[1] :]\nprint(processor.batch_decode(generated_ids, skip_special_tokens=True)[0])","url":"https://huggingface.co/LiquidAI/LFM2.5-VL-3B/blob/35a118d938ce6d123ac2d371649f24a8efb69058/README.md"}],"prerequisites":"See source setup/dependencies; these snippets were inspected, not executed.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["Use multimodal processor and image-text loader; image token count and encoder workspace affect fit.","Grounding/document extraction prompts are task-specific."],"notes":[]},{"id":"lfm25-12-llamacpp-q4","kind":"model","backend":"llama.cpp","repo":"LiquidAI/LFM2.5-1.2B-Instruct","repos":["LiquidAI/LFM2.5-1.2B-Instruct"],"models":["liquidai-lfm2-5-1-2b-instruct"],"url":"https://docs.liquid.ai/lfm/inference/llama-cpp","status":"publisher_or_backend_documented_not_executed_by_researcher","hardware":"CPU/Metal/CUDA according to installed llama.cpp build","steps":[{"code":"uv pip install huggingface-hub\nhf download LiquidAI/LFM2.5-1.2B-Instruct-GGUF lfm2.5-1.2b-instruct-q4_k_m.gguf --local-dir .","url":"https://docs.liquid.ai/lfm/inference/llama-cpp"},{"code":"llama-server -m lfm2.5-1.2b-instruct-q4_k_m.gguf -c 4096 --port 8080","url":"https://docs.liquid.ai/lfm/inference/llama-cpp"},{"code":"llama-cli -m lfm2.5-1.2b-instruct-q4_k_m.gguf -c 4096 --color -i \\\n    --temp 0.1 --top-k 50 --repeat-penalty 1.05","url":"https://docs.liquid.ai/lfm/inference/llama-cpp"}],"sampling":{"temperature":0.1,"top_k":50,"repeat_penalty":1.05},"prerequisites":"See linked source for backend installation/version requirements.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["Exact local q4_k_m filename avoids default-hf artifact ambiguity.","4096 context is documented example, not optimal universal value."],"notes":[]},{"id":"lfm25-12-vllm","kind":"model","backend":"vLLM","repo":"LiquidAI/LFM2.5-1.2B-Instruct","repos":["LiquidAI/LFM2.5-1.2B-Instruct"],"models":["liquidai-lfm2-5-1-2b-instruct"],"url":"https://docs.liquid.ai/lfm/inference/vllm","status":"publisher_or_backend_documented_not_executed_by_researcher","hardware":"unknown","steps":[{"code":"uv pip install -U vllm","url":"https://docs.liquid.ai/lfm/inference/vllm"},{"code":"vllm serve LiquidAI/LFM2.5-1.2B-Instruct","url":"https://docs.liquid.ai/lfm/inference/vllm"}],"prerequisites":"See linked source for backend installation/version requirements.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["Latest vLLM is unpinned; verify loaded architecture and requested context before using fit results."],"notes":[]},{"id":"lfm25-vl3-mlx-dspark","kind":"model","backend":"MLX-VLM","repo":"LiquidAI/LFM2.5-VL-3B","repos":["LiquidAI/LFM2.5-VL-3B"],"models":["liquidai-lfm2-5-vl-3b"],"url":"https://docs.liquid.ai/lfm/inference/mlx","status":"publisher_or_backend_documented_not_executed_by_researcher","hardware":"AppleSilicon unifiedmemory","steps":[{"code":"python -m mlx_vlm.generate \\\n  --model LiquidAI/LFM2.5-VL-3B \\\n  --draft-model LiquidAI/LFM2.5-VL-3B-DSpark \\\n  --draft-block-size 8 \\\n  --image /path/to/image.jpg \\\n  --prompt \"Describe this image.\" \\\n  --max-tokens 256 \\\n  --temperature 0","url":"https://docs.liquid.ai/lfm/inference/mlx"}],"prerequisites":"See linked source for backend installation/version requirements.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["DSpark is a separate draft checkpoint; do not count it as another chat model.","Draft block 8 and additional memory are source configuration; no guaranteed speed multiplier."],"notes":[]},{"id":"falcon-h1r-vllm","kind":"model","backend":"vLLM","repo":"tiiuae/Falcon-H1R-7B","repos":["tiiuae/Falcon-H1R-7B"],"models":["falcon-h1r-7b"],"url":"https://huggingface.co/tiiuae/Falcon-H1R-7B/blob/a6f74bf181389908efd6970878d7ee2b42f5d417/README.md","status":"publisher_documented_not_executed_by_researcher","hardware":"unknown","steps":[{"code":"pip install \"vllm>=0.11.0\"","url":"https://huggingface.co/tiiuae/Falcon-H1R-7B/blob/a6f74bf181389908efd6970878d7ee2b42f5d417/README.md"},{"code":"vllm serve tiiuae/Falcon-H1R-7B \\\n  --tensor-parallel-size 1 \\\n  --data-parallel-size 1 \\\n  --reasoning-parser deepseek_r1","url":"https://huggingface.co/tiiuae/Falcon-H1R-7B/blob/a6f74bf181389908efd6970878d7ee2b42f5d417/README.md"}],"prerequisites":"vLLM>=0.11.0; Transformers alternate requires mamba-ssm[causal-conv1d].","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["Hybrid Mamba/attention state cannot be modeled as ordinary GQA at every layer.","Use deepseek_r1 reasoning parser for vLLM; SGLang names parser deepseek-r1."],"notes":[]},{"id":"falcon-tiny-ollama","kind":"model","backend":"Ollama","repo":"tiiuae/Falcon-H1-Tiny-90M-Instruct","repos":["tiiuae/Falcon-H1-Tiny-90M-Instruct"],"models":["tiiuae-falcon-h1-tiny-90m-instruct"],"url":"https://huggingface.co/tiiuae/Falcon-H1-Tiny-90M-Instruct/blob/e6389502a0b12cd8da894b395ba5bf7436873b16/README.md","status":"publisher_documented_not_executed_by_researcher","hardware":"Any supportedOllamaCPU/GPU backend; exact memory/speed unknown","steps":[{"code":"ollama run hf.co/tiiuae/Falcon-H1-Tiny-90M-Instruct-GGUF:Q8_0","url":"https://huggingface.co/tiiuae/Falcon-H1-Tiny-90M-Instruct/blob/e6389502a0b12cd8da894b395ba5bf7436873b16/README.md"}],"prerequisites":"See source setup/dependencies; these snippets were inspected, not executed.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["ExactQ8_0 GGUF for tiny instruction checkpoint.","Other card examples contain source typos/wrong model ids (SGLang ttiiuae); those are not reproduced as working recipes.","Tiny benchmark or tool competence is not large-model reasoning quality."],"notes":[]},{"id":"k2h-mova-sglang","kind":"model","backend":"SGLang","repo":"IFM/K2-Horizon-MoVA-36B-A4B","repos":["IFM/K2-Horizon-MoVA-36B-A4B"],"models":["k2-horizon-mova-36b-a4b"],"url":"https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B/blob/e9ce717642b2074082875be233c02c729ba47960/README.md","status":"publisher_documented_not_executed_by_researcher","hardware":"2xH200","steps":[{"code":"python3 -m sglang.launch_server \\\n  --model-path IFM/K2-Horizon-MoVA-36B-A4B \\\n  --revision main \\\n  --tp 2 \\\n  --ep 2 \\\n  --dtype bfloat16 \\\n  --attention-backend fa3 \\\n  --json-model-override-args '{\"xllm_source_router_gemm_partitions\":2}' \\\n  --reasoning-parser k2_horizon \\\n  --tool-call-parser k2_horizon \\\n  --host 0.0.0.0 --port 30000","url":"https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B/blob/e9ce717642b2074082875be233c02c729ba47960/README.md"}],"sampling":{"temperature":1,"top_p":0.95,"reasoning_effort":"high"},"prerequisites":"See source setup/dependencies; these snippets were inspected, not executed.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["BF16 TP2 EP2 and FlashAttention 3 validated on 2x H200 in linked cookbook.","xllm_source_router_gemm_partitions override preserves router numerics.","MoVA latent/compressed cache is not ordinary KVheads*head_dim cache."],"notes":[]},{"id":"k2h-mova-vllm","kind":"model","backend":"vLLM","repo":"IFM/K2-Horizon-MoVA-36B-A4B","repos":["IFM/K2-Horizon-MoVA-36B-A4B"],"models":["k2-horizon-mova-36b-a4b"],"url":"https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B/blob/e9ce717642b2074082875be233c02c729ba47960/README.md","status":"publisher_documented_not_executed_by_researcher","hardware":"unknown","steps":[{"code":"vllm serve IFM/K2-Horizon-MoVA-36B-A4B \\\n  --revision main \\\n  --tensor-parallel-size 2 \\\n  --enable-expert-parallel \\\n  --trust-remote-code \\\n  --dtype bfloat16 \\\n  --reasoning-parser k2_horizon \\\n  --tool-call-parser k2_horizon \\\n  --enable-auto-tool-choice","url":"https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B/blob/e9ce717642b2074082875be233c02c729ba47960/README.md"}],"prerequisites":"See source setup/dependencies; these snippets were inspected, not executed.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["Remote/custom code and model-specific reasoning parser; support other K 2 sizes separately.","Revision main in source is mutable; production plan should pin artifact revision."],"notes":[]},{"id":"north-mini-code-vllm","kind":"model","backend":"vLLM","repo":"CohereLabs/North-Mini-Code-1.0","repos":["CohereLabs/North-Mini-Code-1.0"],"models":["north-mini-code"],"url":"https://huggingface.co/CohereLabs/North-Mini-Code-1.0/blob/d11e61a842617a22dc328552fa5bb86231ee4f37/README.md","status":"publisher_documented_not_executed_by_researcher","hardware":"unknown","steps":[{"code":"uv pip install \"git+https://github.com/vllm-project/vllm.git\"\nuv pip install \"cohere_melody>=0.9.0\"","url":"https://huggingface.co/CohereLabs/North-Mini-Code-1.0/blob/d11e61a842617a22dc328552fa5bb86231ee4f37/README.md"},{"code":"vllm serve CohereLabs/North-Mini-Code-1.0 \\\n  -tp 2 \\\n  --max-model-len 320000 \\\n  --tool-call-parser cohere_command4 \\\n  --reasoning-parser cohere_command4 \\\n  --enable-auto-tool-choice","url":"https://huggingface.co/CohereLabs/North-Mini-Code-1.0/blob/d11e61a842617a22dc328552fa5bb86231ee4f37/README.md"}],"sampling":{"temperature":1,"top_p":0.95},"prerequisites":"See source setup/dependencies; these snippets were inspected, not executed.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["Source vLLM build plus cohere_melody>=0.9.0 needed.","Preserve interleaved thinking content across agent steps; strip ping it changes intended behavior.","320000 context and TP2 are source choices requiring full fit."],"notes":[]},{"id":"rnj1-instruct-vllm-tools","kind":"model","backend":"vLLM","repo":"EssentialAI/rnj-1-instruct","repos":["EssentialAI/rnj-1-instruct"],"models":["essentialai-rnj-1-instruct"],"url":"https://huggingface.co/EssentialAI/rnj-1/blob/a48607e59b2c71d297545f160bdb5982c8077b5a/README.md","status":"publisher_documented_not_executed_by_researcher","hardware":"unknown","steps":[{"code":"vllm serve EssentialAI/rnj-1-instruct --enable-auto-tool-choice --tool-call-parser hermes","url":"https://huggingface.co/EssentialAI/rnj-1/blob/a48607e59b2c71d297545f160bdb5982c8077b5a/README.md"}],"prerequisites":"See source setup/dependencies; these snippets were inspected, not executed.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["Cardbase repository documents instruct runtime; recipe explicitly targets EssentialAI/rnj-1-instruct, not base.","Hermes tool parser; do not inherit Gemma 3 sliding attention since Rnj uses global attention."],"notes":[]},{"id":"fara15-4b-vllm","kind":"model","backend":"vLLM","repo":"microsoft/Fara1.5-4B","repos":["microsoft/Fara1.5-4B"],"models":["microsoft-fara1-5-4b"],"url":"https://huggingface.co/microsoft/Fara1.5-4B/blob/776a33ae5b2ad503796a97ae20fdc66f61d2feea/README.md","status":"publisher_documented_not_executed_by_researcher","hardware":"unknown","steps":[{"code":"vllm serve microsoft/Fara1.5-4B \\\n  --dtype bfloat16 \\\n  --max-model-len 262144 \\\n  --limit-mm-per-prompt image=10","url":"https://huggingface.co/microsoft/Fara1.5-4B/blob/776a33ae5b2ad503796a97ae20fdc66f61d2feea/README.md"}],"prerequisites":"See source setup/dependencies; these snippets were inspected, not executed.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["Computer-use task, not generic chat. Publisher recommends MagenticLite execution harness and its model-specific system prompt.","Image limit 10 and 256K context require multimodal memory; no fit claim."],"notes":[]},{"id":"granite42-vllm","kind":"model","backend":"vLLM","repo":"ibm-granite/granite-4.2-8b","repos":["ibm-granite/granite-4.2-8b"],"models":["granite-4.2-8b"],"url":"https://huggingface.co/ibm-granite/granite-4.2-8b/blob/f8de16cdcdbc6c779ca517604e050d82cc119e44/README.md","status":"publisher_documented_not_executed_by_researcher","hardware":"unknown","steps":[{"code":"vllm serve ibm-granite/granite-4.2-8b \\\n    --served-model-name granite-4.2-8b \\\n    --dtype bfloat16 \\\n    --max-model-len 131072 \\\n    --reasoning-parser granite_thinking_parser \\\n    --reasoning-parser-plugin ./granite_thinking_parser.py \\\n    --tool-call-parser qwen3_coder \\\n    --enable-auto-tool-choice","url":"https://huggingface.co/ibm-granite/granite-4.2-8b/blob/f8de16cdcdbc6c779ca517604e050d82cc119e44/README.md"}],"prerequisites":"See source setup/dependencies; these snippets were inspected, not executed.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["Keep tool and reasoning parsers/settings as documented.","Long context still requires fit; 131072 config ceiling is not consumer memory guarantee."],"notes":[]},{"id":"granite-switch-vllm","kind":"model","backend":"vLLM withgranite-switch","repo":"ibm-granite/granite-switch-4.1-8b","repos":["ibm-granite/granite-switch-4.1-8b"],"models":["ibm-granite-granite-switch-4-1-8b"],"url":"https://huggingface.co/ibm-granite/granite-switch-4.1-8b/blob/0d1625d16df2d54cafb6bc5cb0650aec043f77d6/README.md","status":"publisher_documented_not_executed_by_researcher","hardware":"unknown","steps":[{"code":"git clone https://github.com/generative-computing/granite-switch.git\ncd granite-switch\n\n# Pick what you need:\npip install -e \".[compose]\"   # Compose models with adapters\npip install -e \".[hf]\"        # HuggingFace inference\npip install -e \".[vllm]\"      # vLLM inference\npip install -e \".[audio]\"     # Audio input (adds vLLM's audio deps)\npip install -e \".[dev]\"       # Everything","url":"https://huggingface.co/ibm-granite/granite-switch-4.1-8b/blob/0d1625d16df2d54cafb6bc5cb0650aec043f77d6/README.md"},{"code":"pip install -e \".[vllm]\"\n\npython -m vllm.entrypoints.openai.api_server \\\n  --model ibm-granite/granite-switch-4.1-8b \\\n  --port 8000","url":"https://huggingface.co/ibm-granite/granite-switch-4.1-8b/blob/0d1625d16df2d54cafb6bc5cb0650aec043f77d6/README.md"}],"prerequisites":"See source setup/dependencies; these snippets were inspected, not executed.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["Compose dLLM +adapters+speech encoder, not standalone base weight count.","Activation/control tokens and adapter loading essential; base LLM alone does not implement decisions or speech.","Speech encoder is separate ASR cascade, not jointly trained native audio."],"notes":[]},{"id":"granite-swash-transformers","kind":"model","backend":"Transformers","repo":"ibm-granite/granite-swash-2b","repos":["ibm-granite/granite-swash-2b"],"models":["ibm-granite-granite-swash-2b"],"url":"https://huggingface.co/ibm-granite/granite-swash-2b/blob/af1e3227100b61088eead48389ab5409b5d0e39c/README.md","status":"publisher_documented_not_executed_by_researcher","hardware":"unknown","steps":[{"code":"from transformers import AutoModelForCausalLM, AutoTokenizer\nimport torch\n\nmodel = AutoModelForCausalLM.from_pretrained(\n    \"ibm-granite/granite-swash-2b\",\n    dtype=torch.bfloat16,\n    device_map=\"auto\",\n    attn_implementation=\"flash_attention_3\",  # or \"flash_attention_4\" or \"eager\", NOT \"sdpa\"\n)\ntokenizer = AutoTokenizer.from_pretrained(\"ibm-granite/granite-swash-2b\")\n\ninputs = tokenizer(\"The capital of France is\", return_tensors=\"pt\").to(model.device)\nout = model.generate(**inputs, max_new_tokens=50, do_sample=False)\nprint(tokenizer.decode(out[0], skip_special_tokens=True))","url":"https://huggingface.co/ibm-granite/granite-swash-2b/blob/af1e3227100b61088eead48389ab5409b5d0e39c/README.md"}],"prerequisites":"transformers>5.8.0; suitable FlashAttention 3/4 kernel or CPU eager.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["SDPA explicitly unsupported; use flash_attention_3, flash_attention_4 or eager.","Attention sinks and 7 full+17 window 128 layers must be reflected in KV."],"notes":[]},{"id":"olmo-hybrid-transformers","kind":"model","backend":"Transformers","repo":"allenai/Olmo-Hybrid-7B","repos":["allenai/Olmo-Hybrid-7B"],"models":["allenai-olmo-hybrid-7b"],"url":"https://huggingface.co/allenai/Olmo-Hybrid-7B/blob/4f1cc566f9fdf3ce68da2ab6a788a83d89896dcf/README.md","status":"publisher_documented_not_executed_by_researcher","hardware":"unknown","steps":[{"code":"pip install \"transformers>=5.3.0\"","url":"https://huggingface.co/allenai/Olmo-Hybrid-7B/blob/4f1cc566f9fdf3ce68da2ab6a788a83d89896dcf/README.md"},{"code":"from transformers import AutoModelForCausalLM, AutoTokenizer\nolmo = AutoModelForCausalLM.from_pretrained(\"allenai/Olmo-Hybrid-7B\")\ntokenizer = AutoTokenizer.from_pretrained(\"allenai/Olmo-Hybrid-7B\")\nmessage = [\"Language modeling is \"]\ninputs = tokenizer(message, return_tensors='pt', return_token_type_ids=False)\n# optional verifying cuda\n# inputs = {k: v.to('cuda') for k,v in inputs.items()}\n# olmo = olmo.to('cuda')\nresponse = olmo.generate(**inputs, max_new_tokens=100, do_sample=True, top_k=0, temperature=1.0, top_p=0.7)\nprint(tokenizer.batch_decode(response, skip_special_tokens=True)[0])\n>> 'Language modeling is a fundamental task in natural language processing (NLP) that involves predicting the next word in a sequence given   \n  the previous words. It has been widely used in various NLP applications such as machine translation, speech recognition, and text         \n  generation...'","url":"https://huggingface.co/allenai/Olmo-Hybrid-7B/blob/4f1cc566f9fdf3ce68da2ab6a788a83d89896dcf/README.md"}],"prerequisites":"transformers>=5.3.0; quote version constraint in shell toavoidredirection.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["Base completion model; no final hybrid chat/RL checkpoint is listed.","24 DeltaNet layers need recurrent state; 8 full attention layers grow with context."],"notes":[]},{"id":"olmo31-transformers","kind":"model","backend":"Transformers","repo":"allenai/Olmo-3.1-32B-Instruct","repos":["allenai/Olmo-3.1-32B-Instruct"],"models":["allenai-olmo-3-1-32b-instruct"],"url":"https://huggingface.co/allenai/Olmo-3.1-32B-Instruct/blob/ac0587e4a7744a551c059d8cd17ba220bc940dae/README.md","status":"publisher_documented_not_executed_by_researcher","hardware":"unknown","steps":[{"code":"from transformers import AutoModelForCausalLM, AutoTokenizer\nolmo = AutoModelForCausalLM.from_pretrained(\"allenai/Olmo-3.1-32B-Instruct\")\ntokenizer = AutoTokenizer.from_pretrained(\"allenai/Olmo-3.1-32B-Instruct\")\nmessage = [\"Language modeling is \"]\ninputs = tokenizer(message, return_tensors='pt', return_token_type_ids=False)\n# optional verifying cuda\n# inputs = {k: v.to('cuda') for k,v in inputs.items()}\n# olmo = olmo.to('cuda')\nresponse = olmo.generate(**inputs, max_new_tokens=100, do_sample=True, top_k=50, top_p=0.95)\nprint(tokenizer.batch_decode(response, skip_special_tokens=True)[0])\n>> 'Language modeling is  a key component of any text-based application, but its effectiveness...'","url":"https://huggingface.co/allenai/Olmo-3.1-32B-Instruct/blob/ac0587e4a7744a551c059d8cd17ba220bc940dae/README.md"}],"prerequisites":"transformers>=4.57.0; quote version constraint in shell.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["Keep checkpoint identity instruct separate from Think and SFT/DPO flow.","Optional 8bit example in card is not an FP16 artifact or measured speed."],"notes":[]},{"id":"sera14-vllm","kind":"model","backend":"vLLM","repo":"allenai/SERA-14B","repos":["allenai/SERA-14B"],"models":["allenai-sera-14b"],"url":"https://huggingface.co/allenai/SERA-14B/blob/9b585b4126595e471eae9d2c59c8d73dd0fe9e7b/README.md","status":"publisher_documented_not_executed_by_researcher","hardware":"unknown","steps":[{"code":"vllm serve allenai/SERA-14B --port 8001 \\\n            --tensor-parallel-size 4 \\\n            --max-model-len 32768 \\\n            --trust-remote-code \\\n            --enable-auto-tool-choice \\\n            --tool-call-parser hermes \\\n            --enforce-eager \\\n            --seed 42 \\\n            --disable-cascade-attn","url":"https://huggingface.co/allenai/SERA-14B/blob/9b585b4126595e471eae9d2c59c8d73dd0fe9e7b/README.md"}],"prerequisites":"See source setup/dependencies; these snippets were inspected, not executed.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["CodingagentoptimizedforClaudeCodeharness; plain chat benchmarks not same workflow.","Source TP4/32K/FP8 KV configuration is not a requirement nor fit proof for single consumer GPU."],"notes":[]},{"id":"nemotronflash-custom-cuda","kind":"model","backend":"Transformers custommodeling","repo":"nvidia/Nemotron-Flash-3B-Instruct","repos":["nvidia/Nemotron-Flash-3B-Instruct"],"models":["nvidia-nemotron-flash-3b-instruct"],"url":"https://huggingface.co/nvidia/Nemotron-Flash-3B-Instruct/blob/f9f7cb16613a8318a402452fec2f8b7127dbfd99/README.md","status":"publisher_documented_not_executed_by_researcher","hardware":"unknown","steps":[{"code":"from transformers import AutoModelForCausalLM, AutoTokenizer\nimport torch\n\nrepo_name = \"nvidia/Nemotron-Flash-3B-Instruct\"\n\ntokenizer = AutoTokenizer.from_pretrained(repo_name, trust_remote_code=True)\nmodel = AutoModelForCausalLM.from_pretrained(repo_name, trust_remote_code=True)\nmodel = model.cuda().to(torch.bfloat16)\n\nprint('Initializing generation state...')\ngeneration_state = model.init_cuda_graph_generation(\n    max_new_tokens=max_new_tokens,\n    batch_size=1,\n    device='cuda',\n)\n\nprompt = input(\"User:\")\nprompt = \"User: \" + prompt + \"\\nAssistant:\"\ninputs = tokenizer(prompt, return_tensors=\"pt\").to('cuda')\n\nprint(f\"Generating with CUDA graph acceleration...\")\noutputs = model.generate_with_cuda_graph(\n    input_ids=inputs[\"input_ids\"],\n    generation_state=generation_state,\n    max_new_tokens=256,\n    temperature=0,\n    eos_token_id=tokenizer.eos_token_id,\n)\n\nresponse = tokenizer.decode(outputs[0][inputs['input_ids'].shape[1]:], skip_special_tokens=True)\nprint(f\"Response: {response}\")","url":"https://huggingface.co/nvidia/Nemotron-Flash-3B-Instruct/blob/f9f7cb16613a8318a402452fec2f8b7127dbfd99/README.md"},{"code":"outputs = model.generate_with_cache(\n    input_ids=inputs[\"input_ids\"],\n    max_new_tokens=256,\n    temperature=0,\n    eos_token_id=tokenizer.eos_token_id,\n    )","url":"https://huggingface.co/nvidia/Nemotron-Flash-3B-Instruct/blob/f9f7cb16613a8318a402452fec2f8b7127dbfd99/README.md"}],"prerequisites":"Card pins torch<=2.9.1,transformers<=4.56.2,flash-attn<=2.7.3 plus causal-conv1d, mamba-ssm, flash-linear-attention.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["Requires custom CUDA graph/init generation methods; generic generate cannot bepresumedcompatible.","Hybrid architecture fixed state is additional to attention KV; source hardware not universal."],"notes":[]},{"id":"nemotron-diffusion-custom","kind":"model","backend":"Transformers custommodeling","repo":"nvidia/Nemotron-Labs-Diffusion-8B","repos":["nvidia/Nemotron-Labs-Diffusion-8B"],"models":["nvidia-nemotron-labs-diffusion-8b"],"url":"https://huggingface.co/nvidia/Nemotron-Labs-Diffusion-8B/blob/16c67f0560b912e93e0cabb6e0c4f5c3086d95fc/README.md","status":"publisher_documented_not_executed_by_researcher","hardware":"unknown","steps":[{"code":"from transformers import AutoModel, AutoTokenizer\nimport torch\n\nrepo_name = \"nvidia/Nemotron-Labs-Diffusion-8B\"\n\ntokenizer = AutoTokenizer.from_pretrained(repo_name, trust_remote_code=True)\nmodel = AutoModel.from_pretrained(repo_name, trust_remote_code=True)\nmodel = model.cuda().to(torch.bfloat16)\n\nhistory = []\n\nuser_input = input(\"User: \").strip()\nhistory.append({\"role\": \"user\", \"content\": user_input})\n\nprompt = tokenizer.apply_chat_template(history, tokenize=False, add_generation_prompt=True)\nprompt_ids = tokenizer(prompt, return_tensors='pt').input_ids.to(device='cuda')\n\n## Chat in AR Mode\nout_ids, nfe = model.ar_generate(inputs.input_ids, max_new_tokens=512)\n\n## Chat in dLM Mode\nout_ids, nfe = model.generate(prompt_ids, max_new_tokens=512, block_length=32, threshold=0.9, eos_token_id=tokenizer.eos_token_id)\n\n## Chat in Linear Self-Speculation Mode\nout_ids, nfe = model.linear_spec_generate(prompt_ids, max_new_tokens=512, block_length=32, eos_token_id=tokenizer.eos_token_id)\n\ntokenized_out = tokenizer.batch_decode(out_ids[:, prompt_ids.shape[1]:], skip_special_tokens=True)[0]\nprint(f\"Model: {tokenized_out}\")\nprint(f\"[Num Function Eval (NFE)={nfe}]\")","url":"https://huggingface.co/nvidia/Nemotron-Labs-Diffusion-8B/blob/16c67f0560b912e93e0cabb6e0c4f5c3086d95fc/README.md"}],"prerequisites":"transformers>=5.0.0; custom code as shown in card.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["Tri-mode autoregressive/diffusion/block diffusion generation; compare speed only with same mode, steps, block size.","Custom remote-code cache/denoising state is not generic KV."],"notes":[]},{"id":"apertus15-transformers-pinned","kind":"model","backend":"Transformers publisherfork","repo":"swiss-ai/Apertus-v1.5-8B","repos":["swiss-ai/Apertus-v1.5-8B"],"models":["swiss-ai-apertus-v1-5-8b"],"url":"https://huggingface.co/swiss-ai/Apertus-v1.5-8B","status":"publisher_documented_not_executed_by_researcher","hardware":"unknown","steps":[{"code":"pip install \"transformers[torch,vision,audio] @ git+https://github.com/swiss-ai/transformers.git@3797303dda74844e3d1f8977ff5518bb91f818b4\"","url":"https://huggingface.co/swiss-ai/Apertus-v1.5-8B"},{"code":"import torch\nfrom transformers import AutoModelForMultimodalLM, AutoProcessor\n\nMODEL_ID = \"swiss-ai/Apertus-v1.5-8B\"  # or \"swiss-ai/Apertus-v1.5-70B\"\n\nprocessor = AutoProcessor.from_pretrained(MODEL_ID)\nmodel = AutoModelForMultimodalLM.from_pretrained(\n    MODEL_ID, dtype=\"auto\", device_map=\"auto\"\n).eval()\n\n\ndef generate(messages, max_new_tokens=256, **template_kwargs):\n    inputs = processor.apply_chat_template(\n        messages,\n        add_generation_prompt=True,\n        tokenize=True,\n        return_dict=True,\n        return_tensors=\"pt\",\n        **template_kwargs,\n    ).to(model.device)\n    with torch.inference_mode():\n        output_ids = model.generate(**inputs, max_new_tokens=max_new_tokens)\n    return processor.decode(\n        output_ids[0, inputs[\"input_ids\"].shape[-1]:], skip_special_tokens=True\n    )","url":"https://huggingface.co/swiss-ai/Apertus-v1.5-8B"}],"prerequisites":"See source setup/dependencies; these snippets were inspected, not executed.","settings":"All settings in code are publisher/backend examples, not measured optimal values. Values not separatelyjustified must not become global defaults.","limitations":["Pinned transformers 3797303dda74844e3d1f8977ff5518bb91f818b4; upstreaming in progress in card.","Multimodal processor/custom format; auto-generated HF deployment widgets are insufficient verification."],"notes":[]},{"id":"large-kt-deepseek-v4flash0731-rtx5090","kind":"model","backend":"SGLang + KT-Kernel","backend_version":"DSV4-specific Docker or documented source installation; transformers==4.57.1","repo":"deepseek-ai/DeepSeek-V4-Flash-0731","models":["deepseek-v4-flash-0731"],"url":"https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/DeepSeek-V4-Flash.md","status":"source_documented_not_executed_by_researcher","placement":"cpu_experts_gpu_attention","command":"export FLASHINFER_CUDA_ARCH_LIST=12.0a\nexport TORCH_CUDA_ARCH_LIST=\"12.0+PTX\"\n\npython -m sglang.launch_server \\\n  --host 0.0.0.0 --port 30000 \\\n  --model /path/to/models/DeepSeek-V4-Flash-0731 \\\n  --kt-weight-path /path/to/models/DeepSeek-V4-Flash-0731 \\\n  --kt-method MXFP4 \\\n  --kt-num-gpu-experts 10 \\\n  --kt-cpuinfer 60 \\\n  --kt-threadpool-count 2 \\\n  --kt-gpu-prefill-token-threshold 4096 \\\n  --kt-enable-dynamic-expert-update \\\n  --tensor-parallel-size 1 \\\n  --context-length 16384 \\\n  --attention-backend flashinfer \\\n  --mem-fraction-static 0.85 \\\n  --chunked-prefill-size 2048 \\\n  --max-prefill-tokens 2048 \\\n  --max-running-requests 2 \\\n  --watchdog-timeout 1200 \\\n  --disable-shared-experts-fusion \\\n  --trust-remote-code \\\n  --cuda-graph-bs 1 \\\n  --cuda-graph-max-bs 1 \\\n  --disable-radix-cache \\\n  --skip-server-warmup","settings":"Settings are source examples for the stated system/build, not universal optimums.","steps":[],"limitations":[],"notes":["Exact maintainer example. Thread pool counts and CPU threads are workload-specific settings, not evidence of user hardware capacity.","Tutorial reports 20+ decode tokens/s but omits CPU/RAM details; do not use as universal 5090 prediction.","This supports V4-Flash-0731, not V4.1 Flash or 1017. CPU experts are computed on CPU; this differs from NVMe expert streaming to GPU.","Retain Transformers 4.57.1 pin documented by tutorial; Transformers 5.x config incompatibility described."]},{"id":"large-kt-glm52-fp8-tp8","kind":"model","backend":"SGLang + KT-Kernel","backend_version":"Documented KT source + transformers==5.3.0","repo":"zai-org/GLM-5.2","models":["glm-5.2"],"url":"https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/kt-kernel/GLM-5.2-Tutorial.md","status":"source_documented_not_executed_by_researcher","placement":"cpu_experts_multi_gpu","command":"# FP8 Precision\nexport PYTORCH_ALLOC_CONF=expandable_segments:True\nexport SGLANG_ENABLE_JIT_DEEPGEMM=0\n\npython -m sglang.launch_server \\\n  --model-path /path/to/GLM-5.2-FP8 \\\n  --kt-weight-path /path/to/GLM-5.2-FP8 \\\n  --kt-cpuinfer 96 \\\n  --kt-threadpool-count 2 \\\n  --kt-num-gpu-experts 30 \\\n  --kt-method FP8 \\\n  --kt-gpu-prefill-token-threshold 1024 \\\n  --kt-enable-dynamic-expert-update \\\n  --kt-expert-placement-strategy uniform \\\n  --tp-size 8 \\\n  --trust-remote-code \\\n  --host 0.0.0.0 \\\n  --port 8000 \\\n  --mem-fraction-static 0.97 \\\n  --kv-cache-dtype fp8_e4m3 \\\n  --max-total-tokens 4096 \\\n  --max-running-requests 8 \\\n  --attention-backend nsa \\\n  --fp8-gemm-backend cutlass \\\n  --disable-shared-experts-fusion \\\n  --tool-call-parser glm47 \\\n  --reasoning-parser glm45 \\\n  --served-model-name GLM5.2","settings":"Settings are source examples for the stated system/build, not universal optimums.","steps":[],"limitations":[],"notes":["Maintainer validated launch shape, not a consumer 5090 recipe.","FP8 KV + NSA attention + dynamic experts; 30 GPU experts in this launch.","Changing GPU count/CPU threads/memory requires recalibration. No speed claim imported."]},{"id":"large-minicpm5-1b-vllm","kind":"model","backend":"vLLM","backend_version":"vllm>=0.21","repo":"openbmb/MiniCPM5-1B","models":["minicpm5-1b"],"url":"https://huggingface.co/openbmb/MiniCPM5-1B/blob/main/README.md","status":"source_documented_not_executed_by_researcher","placement":"gpu","command":"pip install \"vllm>=0.21\"\nvllm serve openbmb/MiniCPM5-1B --port 8000","settings":"Settings are source examples for the stated system/build, not universal optimums.","steps":[],"limitations":[],"notes":["Publisher minimum version; no hardware-specific speed certification.","SFT/base/midtrain checkpoints are separate weights, not the final RL + OPD assistant."]},{"id":"large-minicpm5-2b-vllm","kind":"model","backend":"vLLM","backend_version":"vllm>=0.21","repo":"openbmb/MiniCPM5-2B","models":["minicpm5-2b"],"url":"https://huggingface.co/openbmb/MiniCPM5-2B/blob/main/README.md","status":"source_documented_not_executed_by_researcher","placement":"gpu","command":"pip install \"vllm>=0.21\"\nvllm serve openbmb/MiniCPM5-2B --port 8000","settings":"Settings are source examples for the stated system/build, not universal optimums.","steps":[],"limitations":[],"notes":["Publisher minimum version; no hardware-specific speed certification.","SFT/base/midtrain checkpoints are separate weights, not the final RL + OPD assistant."]},{"id":"large-minicpm5-2b-llamacpp-f16","kind":"model","backend":"llama.cpp","backend_version":"unknown","repo":"openbmb/MiniCPM5-2B","models":["minicpm5-2b"],"url":"https://huggingface.co/openbmb/MiniCPM5-2B/blob/main/README.md","status":"source_documented_not_executed_by_researcher","placement":"gpu","command":"llama-server -m MiniCPM5-2B-F16.gguf -a MiniCPM5-2B --port 8080 -ngl 99 -c 8192 --jinja","settings":"Settings are source examples for the stated system/build, not universal optimums.","steps":[],"limitations":[],"notes":["Publisher command uses F16 artifact, not an invented Q4 substitution.","Optional DSpark has its own SGLang path; do not assume acceleration on every hardware."]},{"id":"large-mellum21-vllm-tools","kind":"model","backend":"vLLM","backend_version":"unknown","repo":"JetBrains/Mellum2.1-12B-A2.5B-Thinking","models":["mellum2.1-12b-a2.5b-thinking"],"url":"https://huggingface.co/JetBrains/Mellum2.1-12B-A2.5B-Thinking/blob/main/README.md","status":"source_documented_not_executed_by_researcher","placement":"gpu","command":"# Without tool calling\nvllm serve JetBrains/Mellum2.1-12B-A2.5B-Thinking \\\n  --max-model-len 131072 \\\n  --reasoning-parser qwen3\n\n# With tool calling\nvllm serve JetBrains/Mellum2.1-12B-A2.5B-Thinking \\\n  --max-model-len 131072 \\\n  --reasoning-parser qwen3 \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser hermes","settings":"Settings are source examples for the stated system/build, not universal optimums.","steps":[],"limitations":[],"notes":["Card provides separate commands with/without tools.","Card says GGUF and MTP coming soon; not proof of currently supported llama.cpp checkpoint.","GQA 32 query / 4 KV heads with 1024 sliding window in 3 of 4 layers; avoid all 28 layers full-context KV estimate."]},{"id":"large-laguna-xs21-vllm","kind":"model","backend":"vLLM","backend_version":"vllm>=0.21.0","repo":"poolside/Laguna-XS-2.1","models":["poolside-laguna-xs-2-1"],"url":"https://huggingface.co/poolside/Laguna-XS-2.1/blob/main/README.md","status":"source_documented_not_executed_by_researcher","placement":"gpu","command":"pip install 'vllm>=0.21.0'\n\nvllm serve \\\n    --model poolside/Laguna-XS-2.1 \\\n    --tool-call-parser poolside_v1 \\\n    --reasoning-parser poolside_v1 \\\n    --enable-auto-tool-choice \\\n    --served-model-name laguna \\\n    --default-chat-template-kwargs '{\"enable_thinking\": true}'","settings":"Settings are source examples for the stated system/build, not universal optimums.","steps":[],"limitations":[],"notes":["Use poolside_v1 reasoning/tool parsers.","Card says DFlash vLLM support in progress; do not append draft flags until supported."]},{"id":"large-laguna-xs21-llamacpp-pr25165","kind":"model","backend":"llama.cpp fork/PR","backend_version":"PR25165 branch perpublisher","repo":"poolside/Laguna-XS-2.1","models":["poolside-laguna-xs-2-1"],"url":"https://huggingface.co/poolside/Laguna-XS-2.1/blob/main/README.md","status":"source_documented_not_executed_by_researcher","placement":"cpu_default_build","command":"# Build llama.cpp from the PR branch\ngit clone https://github.com/ggml-org/llama.cpp && cd llama.cpp\ngit fetch origin pull/25165/head:laguna && git checkout laguna\ncmake -B build && cmake --build build -j\n\n# Download a GGUF and serve an OpenAI-compatible endpoint\nhuggingface-cli download poolside/Laguna-XS-2.1-GGUF Laguna-XS-2.1-Q4_K_M.gguf --local-dir ~/models/Laguna-XS-2.1-GGUF\n./build/bin/llama-server -m ~/models/Laguna-XS-2.1-GGUF/Laguna-XS-2.1-Q4_K_M.gguf --jinja --port 8000","settings":"Settings are source examples for the stated system/build, not universal optimums.","steps":[],"limitations":[],"notes":["Exact card recipe builds without GGML_CUDA flag; default GPU offload not claimed.","PR branch support is not universal installed llama.cpp/Ollama/LM Studio support."]},{"id":"large-edge0-8b-a1b-ssd","kind":"model","backend":"edge0","backend_version":"publisher edge0 GitHub main at cutoff","repo":"Edge0/Edge0-8B-A1B-preview","models":["edge0-edge0-8b-a1b-preview"],"url":"https://huggingface.co/Edge0/Edge0-8B-A1B-preview/blob/main/README.md","status":"source_documented_not_executed_by_researcher","placement":"trained_router_ssd_expert_streaming","command":"pip install -e 'git+https://github.com/Edge0-AI/edge0.git#egg=edge0[fetch]'\n\n# Download this repository into a local directory\nhuggingface-cli download Edge0/Edge0-8b-a1b-preview --local-dir ./Edge0-8b-a1b-preview\n\n# Run it\nexport EDGE0_8B_MODEL=$PWD/Edge0-8b-a1b-preview\nedge0 chat --name edge0-8b --prompt \"Introduce yourself\"\n\n# Or serve an OpenAI-compatible HTTP API\nedge0 serve --name edge0-8b --port 8083","settings":"Settings are source examples for the stated system/build, not universal optimums.","steps":[],"limitations":[],"notes":["Active-memory footprint is cache only; full checkpoint remains on SSD.","Uses trained routing prediction and LoRA: quality numbers from upstream base must not transfer.","Preview card says complex reasoning and autonomous agent capability currently weak; evaluate task quality separately.","No measured speed copied into this recipe; requires OS/backend compatibility from edge0 docs."]},{"id":"large-edge0-35b-a3b-ssd","kind":"model","backend":"edge0","backend_version":"publisher edge0 GitHub main at cutoff","repo":"Edge0/Edge0-35B-A3B-preview","models":["edge0-edge0-35b-a3b-preview"],"url":"https://huggingface.co/Edge0/Edge0-35B-A3B-preview/blob/main/README.md","status":"source_documented_not_executed_by_researcher","placement":"trained_router_ssd_expert_streaming","command":"pip install -e 'git+https://github.com/Edge0-AI/edge0.git#egg=edge0[fetch]'\n\n# Download this repository into a local directory\nhuggingface-cli download Edge0/Edge0-35b-a3b-preview --local-dir ./Edge0-35b-a3b-preview\n\n# Run it\nexport EDGE0_35B_MODEL=$PWD/Edge0-35b-a3b-preview\nedge0 chat --name edge0-35b --prompt \"Introduce yourself\"\n\n# Or serve an OpenAI-compatible HTTP API\nedge0 serve --name edge0-35b --port 8085","settings":"Settings are source examples for the stated system/build, not universal optimums.","steps":[],"limitations":[],"notes":["Active-memory footprint is cache only; full checkpoint remains on SSD.","Uses trained routing prediction and LoRA: quality numbers from upstream base must not transfer.","Preview card says complex reasoning and autonomous agent capability currently weak; evaluate task quality separately.","No measured speed copied into this recipe; requires OS/backend compatibility from edge0 docs."]},{"id":"large-bonsai2-2b-vision","kind":"model","backend":"PrismML llama.cpp fork","backend_version":"prism-b10776-d00e252 orlater forMTP+quantizedKV onCUDA;branchprism","repo":"prism-ml/Ternary-Bonsai-2-2B-gguf","models":["prism-ml-ternary-bonsai-2-2b-gguf"],"url":"https://huggingface.co/prism-ml/Ternary-Bonsai-2-2B-gguf/blob/main/README.md","status":"source_documented_not_executed_by_researcher","placement":"gpu_or_apple_metal","command":"./bin/llama-server -m Ternary-Bonsai-2-2B-PQ2_0.gguf \\\n  --mmproj Ternary-Bonsai-2-2B-mmproj-Q8_0.gguf -ngl 99 -fa on -c 32768 --port 8080","settings":"Settings are source examples for the stated system/build, not universal optimums.","steps":[],"limitations":[],"notes":["PQ2_0/PTQ1_0 need PrismML fork; upstream only Q2_0 with Hadamard runtime. All three packings contain same weights.","Card recommends Q8_0 projector; BF16 projector may crash concurrent vision requests in current fork.","2B card says drafter slower on everyday prompts; 9B card prefers MTP over DFlash2 for general chat.","MTP 9B file replaces main PQ2_0 file rather than loading both; reserve extra head memory."]},{"id":"large-bonsai2-9b-mtp","kind":"model","backend":"PrismML llama.cpp fork","backend_version":"prism-b10776-d00e252 orlater forMTP+quantizedKV onCUDA;branchprism","repo":"prism-ml/Ternary-Bonsai-2-9B-gguf","models":["prism-ml-ternary-bonsai-2-9b-gguf"],"url":"https://huggingface.co/prism-ml/Ternary-Bonsai-2-9B-gguf/blob/main/README.md","status":"source_documented_not_executed_by_researcher","placement":"gpu_or_apple_metal","command":"./bin/llama-server -m Ternary-Bonsai-2-9B-PQ2_0-MTP-Q8_0.gguf -ngl 99 -fa on \\\n  --spec-type draft-mtp --spec-draft-n-max 2 --spec-draft-n-min 0 --spec-draft-p-min 0","settings":"Settings are source examples for the stated system/build, not universal optimums.","steps":[],"limitations":[],"notes":["PQ2_0/PTQ1_0 need PrismML fork; upstream only Q2_0 with Hadamard runtime. All three packings contain same weights.","Card recommends Q8_0 projector; BF16 projector may crash concurrent vision requests in current fork.","2B card says drafter slower on everyday prompts; 9B card prefers MTP over DFlash2 for general chat.","MTP 9B file replaces main PQ2_0 file rather than loading both; reserve extra head memory."]},{"id":"large-comet60b-vllm-h200-tp4","kind":"model","backend":"vLLM patched nightly","backend_version":"0.28.1rc1.dev628+g2a02f6efe plus bxyu/mtp-fix-try02 commit64edb1d9","repo":"MultiverseComputingCAI/Comet_60B_A12B_BF16","models":["multiversecomputingcai-comet-60b-a12b-bf16"],"url":"https://huggingface.co/MultiverseComputingCAI/Comet_60B_A12B_BF16/blob/main/README.md","status":"source_documented_not_executed_by_researcher","placement":"multi_gpu_full_weights","command":"export VLLM_USE_V2_MODEL_RUNNER=0\nexport VLLM_SSM_CONV_STATE_LAYOUT=DS\n\nvllm serve MultiverseComputingCAI/Comet_60B_A12B_BF16 \\\n  --served-model-name model \\\n  --tensor-parallel-size 4 \\\n  --enable-expert-parallel \\\n  --trust-remote-code \\\n  --max-model-len 262144 \\\n  --gpu-memory-utilization 0.85 \\\n  --kv-cache-dtype fp8 \\\n  --mamba-ssm-cache-dtype float32 \\\n  --enable-prefix-caching \\\n  --no-disable-hybrid-kv-cache-manager \\\n  --block-size 128 \\\n  --mamba-cache-mode align \\\n  --enable-chunked-prefill \\\n  --max-num-seqs 128 \\\n  --max-num-batched-tokens 33920 \\\n  --async-scheduling \\\n  --enable-auto-tool-choice \\\n  --tool-call-parser qwen3_coder \\\n  --reasoning-parser nemotron_v3 \\\n  --speculative-config '{\"method\": \"mtp\", \"num_speculative_tokens\": 3}' \\\n  --limit-mm-per-prompt '{\"image\": 128, \"video\": 1}'","settings":"Settings are source examples for the stated system/build, not universal optimums.","steps":[],"limitations":[],"notes":["Exact publisher benchmark configuration; not necessary minimum GPU count.","Hybrid Mamba state float32, FP8 KV, prefix cache and MTP require both environment variables.","Do not combine MTP with enable-mamba-fine-grained-prefix-cache; publisher observed garbled text in about 1% of long agentic sessions.","Card 1M test confirms startup and needle retrieval, not long-context quality benchmark."]},{"id":"kt1914-qwen3-30b-a3b-instruct-2507-bf16","kind":"model","model":"qwen3-30b-a3b-instruct-2507","url":"https://github.com/kvcache-ai/ktransformers/issues/1914","status":"source_documented_not_executed_by_researcher","hardware":"EPYC 9355 system, 1511 GB total RAM; 8× RTX 5090 installed; each test TP=1","command":"/home/yyj/kt_test/SGLang-f6adb4f-kt1b4665e-x86_64.AppImage --host 0.0.0.0 --port 30002 --model /mnt/data/models/Qwen3-30B-A3B-Instruct-2507 --kt-weight-path /mnt/data/models/Qwen3-30B-A3B-Instruct-2507 --kt-cpuinfer 16 --kt-threadpool-count 1 --kt-num-gpu-experts 28 --kt-method BF16 --attention-backend triton --trust-remote-code --mem-fraction-static 0.85 --chunked-prefill-size 4096 --max-running-requests 1 --max-total-tokens 32000 --enable-mixed-chunk --tensor-parallel-size 1 --disable-shared-experts-fusion","verbatim":true,"settings":"Settings are source examples for the stated system/build, not universal optimums.","steps":[],"limitations":[],"notes":["Source command uses 0.0.0.0; bind 127.0.0.1 for local-only service. No claim of CPU socket count or RAM channel population."],"models":[]},{"id":"kt1914-qwen3.5-122b-a10b-fp8","kind":"model","model":"qwen3.5-122b-a10b","models":["qwen3-5-122b-a10b"],"url":"https://github.com/kvcache-ai/ktransformers/issues/1914","status":"source_documented_not_executed_by_researcher","hardware":"EPYC 9355 system, 1511 GB total RAM; 8× RTX 5090 installed; each test TP=1","command":"/home/yyj/kt_test/SGLang-f6adb4f-kt1b4665e-x86_64.AppImage --host 0.0.0.0 --port 30003 --model /mnt/data/models/Qwen3.5-122B-A10B-FP8 --kt-weight-path /mnt/data/models/Qwen3.5-122B-A10B-FP8 --kt-cpuinfer 16 --kt-threadpool-count 1 --kt-num-gpu-experts 24 --kt-method FP8 --kt-gpu-prefill-token-threshold 2048 --attention-backend triton --fp8-gemm-backend cutlass --trust-remote-code --mem-fraction-static 0.85 --chunked-prefill-size 4096 --max-running-requests 1 --max-total-tokens 32000 --enable-mixed-chunk --tensor-parallel-size 1 --disable-shared-experts-fusion","verbatim":true,"settings":"Settings are source examples for the stated system/build, not universal optimums.","steps":[],"limitations":[],"notes":["Source command uses 0.0.0.0; bind 127.0.0.1 for local-only service. No claim of CPU socket count or RAM channel population."]},{"id":"kt1914-qwen3.5-35b-a3b-fp8","kind":"model","model":"qwen3.5-35b-a3b","models":["qwen3.5-35b-a3b"],"url":"https://github.com/kvcache-ai/ktransformers/issues/1914","status":"source_documented_not_executed_by_researcher","hardware":"EPYC 9355 system, 1511 GB total RAM; 8× RTX 5090 installed; each test TP=1","command":"/home/yyj/kt_test/SGLang-f6adb4f-kt1b4665e-x86_64.AppImage --host 0.0.0.0 --port 30004 --model /mnt/data/models/Qwen3.5-35B-A3B-FP8 --kt-weight-path /mnt/data/models/Qwen3.5-35B-A3B-FP8 --kt-cpuinfer 16 --kt-threadpool-count 1 --kt-num-gpu-experts 64 --kt-method FP8 --kt-gpu-prefill-token-threshold 400 --attention-backend triton --fp8-gemm-backend cutlass --trust-remote-code --mem-fraction-static 0.85 --chunked-prefill-size 4096 --max-running-requests 1 --max-total-tokens 32000 --enable-mixed-chunk --tensor-parallel-size 1 --disable-shared-experts-fusion","verbatim":true,"settings":"Settings are source examples for the stated system/build, not universal optimums.","steps":[],"limitations":[],"notes":["Source command uses 0.0.0.0; bind 127.0.0.1 for local-only service. No claim of CPU socket count or RAM channel population."]},{"id":"kt1914-qwen3-coder-next-bf16","kind":"model","model":"qwen3-coder-next","models":["qwen3-coder-next"],"url":"https://github.com/kvcache-ai/ktransformers/issues/1914","status":"source_documented_not_executed_by_researcher","hardware":"EPYC 9355 system, 1511 GB total RAM; 8× RTX 5090 installed; each test TP=1","command":"/home/yyj/kt_test/SGLang-f6adb4f-kt1b4665e-x86_64.AppImage --host 0.0.0.0 --port 30006 --model /mnt/data/models/Qwen3-Coder-Next --kt-weight-path /mnt/data/models/Qwen3-Coder-Next --kt-cpuinfer 16 --kt-threadpool-count 1 --kt-num-gpu-experts 12 --kt-method BF16 --kt-gpu-prefill-token-threshold 2048 --attention-backend triton --trust-remote-code --mem-fraction-static 0.8 --chunked-prefill-size 16384 --max-running-requests 1 --max-total-tokens 64000 --enable-mixed-chunk --tensor-parallel-size 1 --disable-shared-experts-fusion --kt-enable-dynamic-expert-update","verbatim":true,"settings":"Settings are source examples for the stated system/build, not universal optimums.","steps":[],"limitations":[],"notes":["Source command uses 0.0.0.0; bind 127.0.0.1 for local-only service. No claim of CPU socket count or RAM channel population."]},{"id":"kt1914-qwen3-coder-next-fp8","kind":"model","model":"qwen3-coder-next","models":["qwen3-coder-next"],"url":"https://github.com/kvcache-ai/ktransformers/issues/1914","status":"source_documented_not_executed_by_researcher","hardware":"EPYC 9355 system, 1511 GB total RAM; 8× RTX 5090 installed; each test TP=1","command":"/home/yyj/kt_test/SGLang-f6adb4f-kt1b4665e-x86_64.AppImage --host 0.0.0.0 --port 30007 --model /mnt/data/models/Qwen3-Coder-Next-FP8 --kt-weight-path /mnt/data/models/Qwen3-Coder-Next-FP8 --kt-cpuinfer 16 --kt-threadpool-count 1 --kt-num-gpu-experts 20 --kt-method FP8 --kt-gpu-prefill-token-threshold 2048 --attention-backend triton --fp8-gemm-backend cutlass --trust-remote-code --mem-fraction-static 0.8 --chunked-prefill-size 16384 --max-running-requests 1 --max-total-tokens 64000 --enable-mixed-chunk --tensor-parallel-size 1 --disable-shared-experts-fusion --kt-enable-dynamic-expert-update","verbatim":true,"settings":"Settings are source examples for the stated system/build, not universal optimums.","steps":[],"limitations":[],"notes":["Source command uses 0.0.0.0; bind 127.0.0.1 for local-only service. No claim of CPU socket count or RAM channel population."]},{"id":"kt1914-minimax-m2.5-fp8","kind":"model","model":"minimax-m2.5","models":["minimaxai-minimax-m2-5"],"url":"https://github.com/kvcache-ai/ktransformers/issues/1914","status":"source_documented_not_executed_by_researcher","hardware":"EPYC 9355 system, 1511 GB total RAM; 8× RTX 5090 installed; each test TP=1","command":"/home/yyj/kt_test/SGLang-f6adb4f-kt1b4665e-x86_64.AppImage --host 0.0.0.0 --port 30008 --model /mnt/data/models/MiniMax-M2.5 --kt-weight-path /mnt/data/models/MiniMax-M2.5 --kt-cpuinfer 16 --kt-threadpool-count 1 --kt-num-gpu-experts 8 --kt-method FP8 --kt-max-deferred-experts-per-token 0 --kt-expert-placement-strategy uniform --attention-backend flashinfer --fp8-gemm-backend cutlass --trust-remote-code --mem-fraction-static 0.85 --chunked-prefill-size 4096 --max-running-requests 1 --max-total-tokens 32000 --enable-mixed-chunk --tensor-parallel-size 1 --disable-shared-experts-fusion","verbatim":true,"settings":"Settings are source examples for the stated system/build, not universal optimums.","steps":[],"limitations":[],"notes":["Source command uses 0.0.0.0; bind 127.0.0.1 for local-only service. No claim of CPU socket count or RAM channel population."]},{"id":"dsv4-top4-cpu-numa-modified","kind":"model","model":"deepseek-v4-flash","models":["deepseek-ai-deepseek-v4-flash"],"url":"https://gist.github.com/RockmSockmJesus/30a195ccd9b62e981ec2676a99a57b7e","status":"source_documented_not_executed_by_researcher","command":"#!/usr/bin/env bash\n# DeepSeek-V4-Flash @ ~28 tok/s single-stream, 4-bit, coding quality intact.\n# Box: 1x RTX 5090 (32GB, sm_120) + 2x Xeon Gold 6138 (Skylake-SP, 40 physical cores, AVX-512, NO AMX), 256GB DDR4-2400.\n# Stack: stock kvcache-ai/ktransformers + its sglang (kt-kernel). See README for the one-time convert step.\nset -euo pipefail\n\n# --- edit these three paths ---\nKT_DIR=\"${KT_DIR:-/path/to/ktransformers}\"                 # built ktransformers checkout (contains .venv)\nMODEL=\"${MODEL:-/path/to/dsv4_flash}\"                      # source MIT weights: deepseek-ai/DeepSeek-V4-Flash\nKT_WEIGHTS=\"${KT_WEIGHTS:-/path/to/dsv4_flash_rawint4}\"    # int4 numa-sharded output of convert_cpu_weights_ds4.py\n# ------------------------------\n\ncd \"$KT_DIR\"\nexport CUDA_HOME=/usr/local/cuda\nexport PATH=\"$CUDA_HOME/bin:$PWD/.venv/bin:$PATH\"\nexport FLASHINFER_CUDA_ARCH_LIST=12.0a          # RTX 5090 = sm_120\nexport TORCH_CUDA_ARCH_LIST=\"12.0+PTX\"\nexport SGLANG_DSV4_MODE=2604\nexport SGLANG_DSV4_2604_SUBMODE=2604A           # measured config (upstream docs default to 2604B; both work)\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n\nexec ./.venv/bin/python -m sglang.launch_server \\\n  --host 127.0.0.1 --port 30000 \\\n  --model \"$MODEL\" --kt-weight-path \"$KT_WEIGHTS\" --kt-method AMXINT4 \\\n  --kt-num-gpu-experts 0 --kt-cpuinfer 40 --kt-threadpool-count 2 \\\n  --kt-gpu-prefill-token-threshold 4096 --kt-enable-dynamic-expert-update \\\n  --kt-max-deferred-experts-per-token 3 --kt-expert-placement-strategy frequency \\\n  --tensor-parallel-size 1 --context-length 8192 --attention-backend flashinfer \\\n  --mem-fraction-static 0.70 --chunked-prefill-size 2048 --max-prefill-tokens 2048 \\\n  --max-running-requests 2 --watchdog-timeout 1200 --disable-shared-experts-fusion \\\n  --trust-remote-code --json-model-override-args '{\"num_experts_per_tok\": 4}' \\\n  --cuda-graph-bs 1 --cuda-graph-max-bs 1 --disable-radix-cache --skip-server-warmup \\\n  2>&1 | tee /tmp/kt_server.log\n\n# Notes:\n#   --kt-cpuinfer 40         = ONE thread per PHYSICAL core. Do NOT use hyperthreads (80 -> ~2.7 tok/s, 10x worse).\n#   --kt-num-gpu-experts 0   = CPU-only experts. Adding GPU experts under AMXINT4 was net-negative here (27.5 < 28).\n#   num_experts_per_tok: 4   = top-4 adaptive-k (from top-6). Coding-verified; top-3 and below break coding.\n#   No EAGLE/spec-decode: net-negative on this CPU-bandwidth-bound box (27.2 < 28).\n","settings":"Settings are source examples for the stated system/build, not universal optimums.","steps":[],"limitations":[],"notes":["Research example, not an unchanged-model recommendation; small coding test does not establish general quality equivalence."]},{"id":"jigsaw-v41-direct","kind":"model","backend_version":"JigSawPT/llama.cpp dsv41-porte, 3b6fcfe (same-day server rerun); 8ce0968 CLI tier rerun","model":"deepseek-v4.1-flash","models":["deepseek-v4.1-flash"],"url":"https://github.com/JigSawPT/deepseek-v41-flash-on-5090","status":"source_documented_not_executed_by_researcher","placement":"NVMe paged experts -> pinned RAM tier -> GPU expert cache; GPU expert compute","command":"llama-server -m /path/to/DeepSeek-V4.1-Flash-MXFP4-engram-00001-of-00011.gguf -ngl 99 -c 8192 --moe-stream --moe-stream-cache 18 --moe-stream-l2 72 --moe-stream-direct --reasoning off --host 127.0.0.1 --port 18411 --no-webui","settings":"Settings are source examples for the stated system/build, not universal optimums.","hardware_details":{"gpu_count":1,"gpu":"RTX 5090","reported_vram_gib":31.8,"reported_system_ram_gib":125.7,"system_ram_available_gib":null,"cpu":null,"ram_type":null,"ram_data_rate_mt_s":null,"populated_memory_channels":null,"ssd_model":null,"ssd_capacity_bytes":null,"pcie_negotiated_gen":null,"pcie_negotiated_width":null,"source_url":"https://github.com/JigSawPT/deepseek-v41-flash-on-5090","unknown_reason":"These machine identity/availability fields are absent from the published report; do not borrow them from the separate FreeToken issue."},"artifact_manifest":"research/round2/deepseek_single5090_artifacts.json","steps":[],"limitations":[],"notes":["Target GGUF 11-shard complete manifest required. Optional DSpark is a second artifact, not part of target fit. CPU/SSD/RAM speed unknown. Source above-16384-position candidate-mask correctness not fully tested; do not advertise model maximum as validated recipe context."]},{"id":"jigsaw-v41-dspark","kind":"model","backend_version":"JigSawPT dsv41-porte fork, b10269 base","model":"deepseek-v4.1-flash","models":["deepseek-v4.1-flash"],"url":"https://github.com/JigSawPT/deepseek-v41-flash-on-5090","status":"source_documented_not_executed_by_researcher","command":"llama-server -m /path/to/target-first-shard.gguf -ngl 99 -c 8192 --moe-stream --moe-stream-cache 18 --moe-stream-l2 72 --reasoning off -md /path/to/DeepSeek-V4.1-Flash-DSpark.gguf --spec-type draft-dspark --spec-draft-n-max 2 --spec-draft-n-cpu-moe 3 --host 127.0.0.1","settings":"Settings are source examples for the stated system/build, not universal optimums.","steps":[],"limitations":[],"notes":["Measured mixed-prompt DSpark gain was neutral or negative; optional draft consumes extra memory and does not make resident-cache rates general fresh-content performance."]},{"id":"devin-v41-fast","kind":"model","backend_version":"vLLM 8c1d1c2974ee42757ee2e93cc898932edfd9d265 plus repository patches; torch2.13 cu130; CUDA13.2; FlashInfer0.6.18.post1","model":"deepseek-v4.1-flash","models":["deepseek-v4.1-flash"],"url":"https://github.com/devin-lai/DeepSeek-V4.1-Flash-Accel","status":"source_documented_not_executed_by_researcher","command":"HOST=127.0.0.1 MODEL=/data/models/DeepSeek-V4.1-Flash PRESET=v41-flash-fast deploy/serve.sh","prerequisites":"Use repository pinned setup and verified full source checkpoint. 8× RTX5090 D, dual Xeon 6530,503GiB host reference; physical RAM minimum is not established by cgroup cap.","settings":"Settings are source examples for the stated system/build, not universal optimums.","steps":[],"limitations":[],"notes":["Expert GEMM runs on GPU, including host-backed pages; this is not CPU expert computation. Placement profile is domain dependent."]},{"id":"devin-v41-fast-placed","kind":"model","backend_version":"vLLM 8c1d1c2974ee42757ee2e93cc898932edfd9d265 plus repository patches; torch2.13 cu130; CUDA13.2; FlashInfer0.6.18.post1","model":"deepseek-v4.1-flash","models":["deepseek-v4.1-flash"],"url":"https://github.com/devin-lai/DeepSeek-V4.1-Flash-Accel","status":"source_documented_not_executed_by_researcher","command":"HOST=127.0.0.1 MODEL=/data/models/DeepSeek-V4.1-Flash PRESET=v41-flash-fast-placed deploy/serve.sh","prerequisites":"Use repository pinned setup and verified full source checkpoint. 8× RTX5090 D, dual Xeon 6530,503GiB host reference; physical RAM minimum is not established by cgroup cap.","settings":"Settings are source examples for the stated system/build, not universal optimums.","steps":[],"limitations":[],"notes":["Expert GEMM runs on GPU, including host-backed pages; this is not CPU expert computation. Placement profile is domain dependent."]},{"id":"template-llama-cpu","kind":"parameterized_backend_template","backend":"llama.cpp","url":"https://github.com/ggml-org/llama.cpp/blob/1268edac579110aeb1407f7928f53193ed4e28d4/tools/server/README.md","status":"syntax_documented_template_not_executed","placement":"cpu_only","template":"llama-server -m <MODEL.gguf> -ngl 0 -c <CONTEXT> -t <CPU_THREADS> -ctk f16 -ctv f16","settings":"Placeholders must be bound from a verified artifact/architecture/hardware plan; no measured-optimal settings asserted.","steps":[],"limitations":["Use a supported complete GGUF artifact and resolve RAM/cache/workspace. Tune threads against physical cores/NUMA using measured data.","Source default mmap is not a proof all weights are resident."],"notes":[],"models":[]},{"id":"template-llama-gpu","kind":"parameterized_backend_template","backend":"llama.cpp","url":"https://github.com/ggml-org/llama.cpp/blob/1268edac579110aeb1407f7928f53193ed4e28d4/tools/server/README.md","status":"syntax_documented_template_not_executed","placement":"full_gpu","template":"llama-server -m <MODEL.gguf> -ngl all -c <CONTEXT> -fa on -ctk <SUPPORTED_K_TYPE> -ctv <SUPPORTED_V_TYPE>","settings":"Placeholders must be bound from a verified artifact/architecture/hardware plan; no measured-optimal settings asserted.","steps":[],"limitations":["Only after all-device memory ledger fits, including any mmproj/draft/lookup dependencies.","Use the backend actually built: CUDA/HIP/Vulkan/SYCL/Metal, with separate performance evidence."],"notes":[],"models":[]},{"id":"template-llama-partial","kind":"parameterized_backend_template","backend":"llama.cpp","url":"https://github.com/ggml-org/llama.cpp/blob/1268edac579110aeb1407f7928f53193ed4e28d4/tools/server/README.md","status":"syntax_documented_template_not_executed","placement":"gpu_layers_cpu_layers","template":"llama-server -m <MODEL.gguf> -ngl <GPU_LAYER_COUNT> -c <CONTEXT> -t <CPU_THREADS> -ctk <SUPPORTED_K_TYPE> -ctv <SUPPORTED_V_TYPE>","settings":"Placeholders must be bound from a verified artifact/architecture/hardware plan; no measured-optimal settings asserted.","steps":[],"limitations":["Layer counts do not directly equal a weight-byte percentage. Inspect load log allocation and tune GPU layer count at fixed context."],"notes":[],"models":[]},{"id":"template-llama-cpu-experts","kind":"parameterized_backend_template","backend":"llama.cpp","url":"https://github.com/ggml-org/llama.cpp/blob/1268edac579110aeb1407f7928f53193ed4e28d4/tools/server/README.md","status":"syntax_documented_template_not_executed","placement":"cpu_experts_gpu_attention","template":"llama-server -m <MOE_MODEL.gguf> -ngl all --n-cpu-moe <CPU_EXPERT_LAYER_COUNT> -c <CONTEXT> -t <CPU_THREADS> -fa on -ctk <SUPPORTED_K_TYPE> -ctv <SUPPORTED_V_TYPE>","settings":"Placeholders must be bound from a verified artifact/architecture/hardware plan; no measured-optimal settings asserted.","steps":[],"limitations":["--n-cpu-moe counts first N layers, not N routed experts. CPU GEMV/GEMM and host bandwidth govern the expert path.","Do not use for dense models; choose --n-cpu-ffn only where its separate supported path is intended."],"notes":[],"models":[]},{"id":"template-llama-multigpu-layer","kind":"parameterized_backend_template","backend":"llama.cpp","url":"https://github.com/ggml-org/llama.cpp/blob/1268edac579110aeb1407f7928f53193ed4e28d4/docs/multi-gpu.md","status":"syntax_documented_template_not_executed","placement":"multi_gpu_layer","template":"llama-server -m <MODEL.gguf> -ngl all --split-mode layer --tensor-split <RELATIVE_DEVICE_WEIGHTS> -c <CONTEXT> -fa on","settings":"Placeholders must be bound from a verified artifact/architecture/hardware plan; no measured-optimal settings asserted.","steps":[],"limitations":["Validate each GPU allocation and interconnect. Single-stream decode does not receive summed GPU bandwidth.","Do not assume P2P works; inspect backend topology/logs."],"notes":[],"models":[]},{"id":"template-llama-multigpu-tensor","kind":"parameterized_backend_template","backend":"llama.cpp","url":"https://github.com/ggml-org/llama.cpp/blob/1268edac579110aeb1407f7928f53193ed4e28d4/docs/multi-gpu.md","status":"syntax_documented_template_not_executed","placement":"multi_gpu_tensor","template":"llama-server -m <SUPPORTED_MODEL.gguf> -ngl all --split-mode tensor --tensor-split <RELATIVE_DEVICE_WEIGHTS> --fit off -c <CONTEXT> -fa on -ctk f16 -ctv f16","settings":"Placeholders must be bound from a verified artifact/architecture/hardware plan; no measured-optimal settings asserted.","steps":[],"limitations":["Experimental; exclude unsupported model architectures, quantizations and builds.","Audited tensor split requires nonquantized KV and has no automatic --fit support. NCCL/RCCL and P2P support must be checked."],"notes":[],"models":[]},{"id":"template-vllm-uva","kind":"parameterized_backend_template","backend":"vLLM","url":"https://docs.vllm.ai/en/latest/cli/serve/","status":"syntax_documented_template_not_executed","placement":"gpu_weight_stream_from_ram","template":"vllm serve <EXACT_HF_REPO> --revision <REVISION> --max-model-len <CONTEXT> --tensor-parallel-size <GPU_COUNT> --offload-backend uva --cpu-offload-gb <GIB_PER_GPU>","settings":"Placeholders must be bound from a verified artifact/architecture/hardware plan; no measured-optimal settings asserted.","steps":[],"limitations":["GPU computes CPU-backed weights through UVA; this is not CPU expert computation.","Budget host memory across every rank, plus staging and KV. Confirm model architecture and native quant kernels on exact GPU/driver."],"notes":[],"models":[]},{"id":"template-vllm-prefetch","kind":"parameterized_backend_template","backend":"vLLM","url":"https://docs.vllm.ai/en/latest/cli/serve/","status":"syntax_documented_template_not_executed","placement":"gpu_weight_stream_from_ram","template":"vllm serve <EXACT_HF_REPO> --revision <REVISION> --max-model-len <CONTEXT> --offload-backend prefetch --offload-group-size <LAYERS_PER_GROUP> --offload-num-in-group <OFFLOADED_LAYERS_PER_GROUP> --offload-prefetch-step <LOOKAHEAD>","settings":"Placeholders must be bound from a verified artifact/architecture/hardware plan; no measured-optimal settings asserted.","steps":[],"limitations":["Different scheduler from UVA; prefetch memory and eligible parameter selection must be accounted. Values are plan-specific and must be benchmarked."],"notes":[],"models":[]},{"id":"template-mlx-kv","kind":"parameterized_backend_template","backend":"MLX-LM","url":"https://github.com/ml-explore/mlx-lm/blob/93e6d4a6447b3080861985947f0820e76fcb957d/mlx_lm/generate.py","status":"syntax_documented_template_not_executed","placement":"apple_unified","template":"mlx_lm.generate --model <PINNED_LOCAL_MLX_DIRECTORY> --prompt <PROMPT> --max-tokens <MAX_OUTPUT_TOKENS> --kv-bits <SUPPORTED_KV_BITS> --kv-group-size <GROUP_SIZE> --prefill-step-size <CHUNK_TOKENS>","settings":"Placeholders must be bound from a verified artifact/architecture/hardware plan; no measured-optimal settings asserted.","steps":[],"limitations":["Not every architecture supports quantized cache. Omitting --max-kv-size avoids silently imposing a rotating-cache retention cap.","HF model ID/revision should first resolve to an exact local snapshot. GPU and CPU share RAM."],"notes":[],"models":[]},{"id":"template-ollama-explicit-context","kind":"parameterized_backend_template","backend":"Ollama","status":"documented_template_not_executed","steps":[{"code":"FROM <SUPPORTED_GGUF_OR_PINNED_MODEL>\nPARAMETER num_ctx <CONTEXT>"},{"code":"ollama create localfit-plan -f Modelfile\nollama run localfit-plan\nollama ps"}],"limitations":["Use an exact digest/file; upstream tags can move.","Ollama documentation gives conflicting context defaults, so set explicit context. CPU/GPU PRO CESSOR split is not an exact byte count or performance prediction.","Record environment cache, flash-attention and concurrency settings; model-specific expert-offload controls may not be exposed."],"notes":[],"models":[]},{"id":"template-lmstudio-load","kind":"parameterized_rest_template","backend":"LM Studio","url":"https://lmstudio.ai/docs/developer/rest/load","status":"documented_template_not_executed","steps":[],"limitations":["Bind placeholders to real types; source example settings are not proof of architecture support.","Set GPU and CPU expert layer ratio through the supported UI/SDK and capture actual engine/load logs. Do not invent REST fields for them.","numExperts changes active expert routing and output quality; it is not CPU offload placement."],"notes":[],"models":[]}]}