LocalFit
Submit a benchmark

NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16

nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16

Released2026-10-16Size4B denseRoleAssistantLicenseopenmdw-1.1Context256K config max

Where it runs 8K context · ≥ 5 t/s floor

SetupBest planMemoryGenerate
8 GB GPU + 32 GB RAMQ5_K_Mllama.cpp CUDAFull GPUKV unknown~45–65 t/s Estimated
12 GB GPU + 32 GB RAMQ5_K_Mllama.cpp CUDAFull GPUKV unknown~75–80 t/s Estimated
16 GB GPU + 64 GB RAMQ5_K_Mllama.cpp CUDAFull GPUKV unknown~155–190 t/s Estimated
24 GB GPU + 64 GB RAMQ5_K_Mllama.cpp CUDAFull GPUKV unknown~157–197 t/s Estimated
32 GB GPU + 128 GB RAMQ5_K_Mllama.cpp CUDAFull GPUKV unknown~246–393 t/s Estimated
2× 24 GB GPUs + 128 GB RAMQ5_K_Mllama.cpp CUDAFull GPUKV unknown~143–185 t/s Estimated
96 GB workstation GPU + 256 GB RAMQ5_K_Mllama.cpp CUDAFull GPUKV unknown~252–396 t/s Estimated
Mac, 24 GBQ5_K_Mllama.cpp MetalUnified memoryKV unknown~53–63 t/s Estimated
Mac, 48 GBQ5_K_Mllama.cpp MetalUnified memoryKV unknown~53–63 t/s Estimated
Mac, 128 GBQ5_K_Mllama.cpp MetalUnified memoryKV unknown~86–131 t/s Estimated
Mac, 512 GBQ5_K_Mllama.cpp MetalUnified memoryKV unknown~96–147 t/s Estimated
DGX Spark 128 GBQ5_K_Mllama.cpp CUDAUnified memoryKV unknown~57–63 t/s Estimated
Ryzen AI Max+ 395, 128 GBQ5_K_Mllama.cpp VulkanUnified memoryKV unknown~38–55 t/s Estimated
CPU only, 64 GB DDR5-5600 dual channelQ5_K_Mllama.cpp CPUCPU onlyKV unknown~5.5–6.6 t/s Estimated extrapolated

Same plan engine as the planner, with its default reserves and RAM. How plans are chosen.

Files exact Hugging Face bytes

ArtifactSizeFilesRevisionSource
Q5_K_M
nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-GGUF
3.42 GiB10963c80007HF files
Native safetensors BF16
nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16
7.40 GiB1b366b87f76HF files

Not found in the representative quant search (not proven unavailable): GGUF FP16, GGUF Q8_0, GGUF Q4_K_M, MLX 8-bit, MLX 4-bit, EXL3, AWQ, GPTQ. Complete selected file sets; projectors, MTP/draft and lookup files are listed separately and budgeted only by plans that use them. File bytes are not allocated memory.

Quality

No Artificial Analysis evaluation mapped to this checkpoint.

Arena text rating

Unknown

Publisher-reported benchmarks

None extracted from the model card.

AA and Arena describe the hosted model in the stated mode, not each quantized file; scores are never inherited by base, pruned, abliterated or quantized variants and never averaged together.

How to run it

vLLM · NVIDIA CUDA; exact device/memory unknown

publisher-documented, not executed source

VLLM_USE_PRECOMPILED=1 pip install "git+https://github.com/vllm-project/vllm.git@refs/pull/60149/head"
vllm serve --model nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-BF16 \
    --hf-overrides '{"n_recurrent_loops": 2, "recurrent_kv_cache_mode": "per_loop"}' \
    --max-model-len 262144 \
    --max-num-seqs 128 \
    --enable-prefix-caching \
    --async-scheduling \
    --reasoning-parser nemotron_v3 \
    --tool-call-parser qwen3_coder \
    --enable-auto-tool-choice

Prerequisites: See source setup/dependencies; these snippets were inspected, not executed.

Sampling: temperature=1 · top_p=0.95

  • Support under review in PR60149; not a verified stable-release recipe.
  • Two loops reuse weights but per_loop caches grow with loop count. last_loop is a distinct cache mode.
  • Card 256K context/max_num_seqs128 is an example, not a consumer memory-fit guarantee.

llama.cpp · Apple Silicon ARM64 Metal; macOS26 SDK specified by publisher

publisher-documented, not executed source

brew install cmake openssl@3

git init llama.cpp-dory
cd llama.cpp-dory
git remote add origin https://github.com/brianchmielNV/llama.cpp.git
git fetch --depth 1 origin 98a74c97ea09ac72f6ceda1559808fc0a2f447ce
git checkout --detach FETCH_HEAD

HF_UI_VERSION=b11551 cmake -S . -B build \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_OSX_ARCHITECTURES=arm64 \
    -DCMAKE_OSX_SYSROOT="/Library/Developer/CommandLineTools/SDKs/MacOSX26.sdk" \
    -DGGML_METAL=ON -DGGML_METAL_EMBED_LIBRARY=ON \
    -DGGML_BLAS=ON -DGGML_BLAS_VENDOR=Apple \
    -DLLAMA_BUILD_SERVER=ON \
    -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_EXAMPLES=OFF -DLLAMA_BUILD_APP=OFF \
    -DLLAMA_BUILD_UI=OFF -DLLAMA_USE_PREBUILT_UI=ON \
    -DLLAMA_OPENSSL=ON -DOPENSSL_ROOT_DIR="$(brew --prefix openssl@3)"
cmake --build build --parallel 4 --target llama-server
Download dory-q5_k_m.gguf from https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4B-GGUF and place it in llama.cpp-dory/.
echo "96080be49ffd67b916018b7bdef8c484762576fb52780e10ab98ac4978a660e0  dory-q5_k_m.gguf" | shasum -a 256 -c -
./build/bin/llama-server \
    -m ./dory-q5_k_m.gguf \
    --alias dory \
    --host 127.0.0.1 --port 8080 \
    -ngl 999 \
    --ctx-size 4096 --parallel 1 \
    --batch-size 512 --ubatch-size 512 \
    --cache-type-k f16 --cache-type-v f16 \
    --flash-attn off \
    --no-context-shift \
    --jinja --reasoning on \
    --temp 1.0 --top-p 0.95 --top-k 0 --min-p 0 --repeat-penalty 1 \
    --override-kv "dory.num_loops=int:2,dory.recurrent_kv_cache_mode=str:per_loop"

Prerequisites: See source setup/dependencies; these snippets were inspected, not executed.

Sampling: temperature=1 · top_p=0.95 · top_k=0 · min_p=0 · repeat_penalty=1

  • Pinned fork 98a74c97ea09ac72f6ceda1559808fc0a2f447ce; upstream support PR30306.
  • Card command uses Q5_K_M, 4096 context, batch 512, FP16 KV, flash attention off, per_loop. Do not silently assume FA or quantized KV support.
  • Changing recurrent_kv_cache_mode to last_loop saves recurrent cache but changes execution configuration.

Recipes are source examples (settings are the publisher’s or author’s, not measured optima) and were not executed by LocalFit. A downloadable file does not prove your backend build supports the architecture.

Reported speeds 0

No reported measurement yet. Submit one.