Tutorial on running NVIDIA Dory 4B with llama.cpp released

MarkTechPost released a tutorial demonstrating how to run NVIDIA Nemotron Labs' Dory 1.0 4B reasoning model with llama.cpp. The model's intermediate layers form a shared block that can be applied repeatedly 1 to 4 times per token without increasing the parameter count. The tutorial says that because llama.cpp's support for Dory is still an unmerged draft PR, no released binary can load the model, so llama-server must be compiled from the commit specified in the model card. The tutorial also points out two pitfalls: the file name dory-q5_k_m.gguf in the model card command does not match the repository's actual file name, so copying it verbatim will fail; and the loop count built into the model file header is 4, while the model card recommends 2, so failing to override that setting will run the most expensive configuration. The tutorial lists claims that cannot be verified in its environment, such as the benchmark table, 256K context, and NVFP4 pretraining, as untested.

AI overview based on source reports · Updated 10 hr ago