Nemotron 3.5 Bolt 74B-A9B in EXL3, 6.0 bits per weight

These are EXL3 weights of NVIDIA's Nemotron 3.5 Bolt 74B-A9B at 6.0 bits per weight: 6.00 bits on average over the text weights, 56.6 GB. We converted them from NVIDIA's BF16 checkpoint, nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-BF16 at revision bca0a8cb, with exllamav3 1.6.2 and a patch for Bolt's layer layout (in convert/). Every quantized projection is at 6.0 bits: the routed experts and the Mamba2, attention, latent and shared-expert projections alike. The output head is at 8 bits. The embedding stays in BF16. The RADIO image tower and its projector are in the files unchanged, in BF16 (1.7 GB); TandemLLM serves text only for now and does not load them.

We serve these weights with TandemLLM, our inference engine for one DGX Spark. Bolt is 88 layers: 40 Mamba2, 40 mixture-of-experts with 240 to 496 experts each (9 a token, plus one shared expert) and 8 attention layers, 262,144 tokens of context.

Quality of every level

We compared each pack with BF16 on 8 rows of wikitext-2, 1,024 tokens each (8,192 positions). The KL, top-1 and perplexity columns are from TandemLLM's engine against BF16's logits on the same tokens. BF16's perplexity on these rows is 5.97. The model_diff columns are exllamav3's eval/model_diff.py on 100 rows of 2,048 tokens, where BF16's perplexity is 5.66; that run is in quality/ of each repository. GB counts the text weights only.

level	bpw	GB	KL	top-1	ppl	KL (model_diff)	top-1 (model_diff)
2.0	2.19	21.7	0.170	84.0 %	6.81	0.0881	87.6 %
2.5	2.64	25.8	0.094	87.3 %	6.34	0.0492	90.6 %
3.0	3.09	30.0	0.061	89.7 %	6.21	0.0321	92.2 %
3.5	3.55	34.1	0.043	91.1 %	6.12	0.0236	93.3 %
4.0	4.00	38.2	0.034	92.7 %	6.10	0.0197	93.8 %
5.0	5.00	47.4	0.016	94.8 %	6.00	0.0084	96.0 %
6.0	6.00	56.6	0.009	96.2 %	5.98	0.0047	97.1 %
NVIDIA IQ2 GGUF	2.38	22.0	0.259	79.6 %	7.48

Below 4.0 we keep every projection outside the routed experts at 4 bits. On the 2.0 pack that change took KL from 0.379 to 0.170. From 4.0 up every projection is at the level.

Speed on a DGX Spark

TandemLLM 0.10.0 on one DGX Spark (GB10, 128 GB), the box running nothing else. One request at a time, 512 greedy tokens, thinking off, a 32,768-token window, a fresh server for every run. Every level ran three plain runs. 2.0, 4.0 and NVIDIA's file also ran three runs with NVIDIA's LiLiCorr drafter plus TandemLLM's lookup (StairCut picks the block size), alternating with the plain ones; for the other levels the drafted speed is not measured, see 2.0 and 4.0. Each cell is the mean and ± half the range of the runs. Plain is the mean over the five asks. Edit and copy are asks whose answer repeats much of the prompt. On every ask, the text with LiLiCorr and the lookup was the same as the plain text. On 4.0 and NVIDIA's file, TandemLLM 0.10.0 gets no speed from LiLiCorr and the lookup: its step-size rule (StairCut) keeps plain steps on every ask, so those cells are plain decoding. We have not found the cause yet; TandemLLM 0.10.0's profile for NVIDIA's file drafts with the lookup alone.

tok/s	plain	LiLiCorr, prose	chat	code	edit	copy
2.0	43.3 ± 0.05	65.7 ± 0.2	69.4 ± 0.4	102.8 ± 0.2	180.5 ± 10.8	244.7 ± 4.2
2.5	42.1 ± 0.1	-	-	-	-	-
3.0	41.7 ± 0.2	-	-	-	-	-
3.5	40.6 ± 0.2	-	-	-	-	-
4.0	40.5 ± 0.1	40.4 ± 0.1	40.5 ± 0	40.5 ± 0	40.3 ± 0.05	40.3 ± 0.05
5.0	34.1 ± 0.05	-	-	-	-	-
6.0	29.2 ± 0	-	-	-	-	-
NVIDIA IQ2 GGUF	27.8 ± 0.1	27.2 ± 0.4	27.6 ± 0.3	27.4 ± 0.5	27.6 ± 0.05	27.5 ± 0.1

The 2.0 card compares our 2.0 pack with NVIDIA's 2-bit GGUF in detail.

How to run it

TandemLLM v0.10.0 runs every level (release notes). On a DGX Spark, install it in setup mode, then switch to this level:

curl -fsSLO https://raw.githubusercontent.com/0xBakeer/TandemLLM/v0.10.0/install.sh
bash install.sh --setup --yes
~/TandemLLM/run/ops/use-profile.sh nemotron-bolt-exl3-6.0

use-profile.sh downloads this repository at a pinned commit, checks each file against its sha256, fetches NVIDIA's LiLiCorr drafter and starts the engine on port 8000. It needs no Hugging Face token. A start needs about 53 GiB for the weights. All seven levels are in the collection.

The files are in the EXL3 format, but stock exllamav3 1.6.2 does not know Bolt's layer layout and will not load them. exllamav3 1.6.2 with the patch in convert/ loads them; we measured them with it. We have not tested other EXL3 loaders.

How we made it

convert/exllamav3-1.6.2-bolt.patch applies to exllamav3 v1.6.2 (commit 4c84273). It reads Bolt's per-layer block table and its Omni wrapper, gives each MoE layer its own expert count, and copies the image tower unquantized. convert/README.md has the commands. Each pack used exllamav3's built-in calibration (250 rows of 2,048 tokens) and the mul1 codebook; quality/convert.txt is the log.

License

These weights are a quantization of NVIDIA's model and carry its license, the OpenMDW License Agreement 1.1, in LICENSE. NOTICE says where the files come from and what we changed. The patch in convert/ changes exllamav3, which is MIT-licensed. Everything else about the model, including its intended use and limits, is in NVIDIA's model card.

Credits

NVIDIA made Nemotron 3.5 Bolt, its BF16 checkpoint, the IQ2 GGUF and the LiLiCorr drafter (NVFP4, GGUF). turboderp made exllamav3 and the EXL3 format.