czl/CLM-v0.1-8B-GGUF

CLM-v0.1-8B — GGUF encoder (Q4_K_M, Q5_K_M, Q6_K, Q8_0)

The encoder half of Contrastive-LM/CLM-v0.1-8B, quantised for llama.cpp. CLM is a ranker, not a generator: a frozen Qwen3-8B encoder with last-token pooling feeding two frozen projection heads. This repo is the encoder. The heads are not quantised and not re-shipped — clm-serve already fetches CLM_v0.1-8B.pt from the parent repo into ~/.cache/clm/.

What is in this repo

file	what
Qwen3-8B-BF16.gguf	the reference. The unquantised encoder straight from convert_hf_to_gguf.py --outtype bf16, published so every number below is checkable and so the baseline is available without re-deriving it
Qwen3-8B-Q8_0.gguf	8-bit K-quant, the closest to lossless type in llama.cpp — 8.71 GB
Qwen3-8B-Q6_K.gguf	6-bit K-quant (Q6_K is the only 6-bit K type; there is no _M/_S) — 7.03 GB
Qwen3-8B-Q5_K_M.gguf	5-bit mixture at 6.09 bpw — the same perfect decisive bucket as Q6_K in 792 MB less — 6.24 GB
Qwen3-8B-Q4_K_M.gguf	4-bit mixture: Q4_K baseline, attn_v/ffn_down promoted to Q6_K on the first and last eighth of layers and every third layer between, fused attn_qkv at Q5_K — 5.49 GB
imatrix.gguf	the importance matrix used for every quant below, so the quantisation is reproducible
clm_config.json	the pooling / embedding / scale contract, which no GGUF key expresses on its own

Every quantised file has last-token pooling baked in (pooling_type = 3), so no runtime flag is needed. token_embd.weight and lm_head are held at 8-bit in all of them.

How to use

# 1. encoder: last-token pooled embeddings on the OpenAI-compatible endpoint.
#    -c is the TOTAL context and is split across the -np slots, so 8 x 2048 is what
#    you want if each request may use the full 2048-token window.
llama-server -m Qwen3-8B-Q8_0.gguf --embedding --pooling last -c 16384 -np 8 -ngl 30 --port 8090

# 2. CLM itself -- unchanged, it already speaks this endpoint
clm-serve

clm-serve fetches the heads into ~/.cache/clm/ on first run. See the parent model card for the full API.

Evaluation

Agreement against a bf16 GGUF reference of the same encoder in the same runtime, over 23,926 scored System One questions. The corpus is built from CLM's own data: the two cases the parent model card publishes, 23,858 T-Rex decisions replayed from the shipped deterministic runner and labelled by its physics planner, and a long-document stratum. Each model is scored end to end with its own vectors through the real head stack.

variant	cos min	cos mean	top-1	top-1 (decisive)	planner acc. Δ	verdict
BF16	–	–	1.0000 (ref)	1.0000 (ref)	–	reference
Q8_0	0.99294	0.99986	0.9786	1.0000	+0.46 pts	yes
Q6_K	0.98539	0.99937	0.9278	1.0000	-2.88 pts	usable
Q5_K_M	0.92198	0.99870	0.8822	1.0000	-2.05 pts	usable
Q4_K_M	0.91016	0.99690	0.8429	0.9958	-6.63 pts	no

Pass line. top-1 >= 1.0000 — the measured bf16-vs-bf16 noise floor of this corpus in this runtime — and top-1 (decisive) >= 0.995, where decisive means the reference's own top-1 led by more than 1 nat. A third condition applies: planner accuracy Δ, the change in accuracy against the T-Rex physics planner's label. A variant that holds every decisive decision but still loses measurable accuracy is marked usable rather than recommended — the Q6_K result below is exactly that case.

Why the margins are split. The answer is argmax(scale · cos) with scale = exp(logit_scale) = 100.0 — HeadPair._load clamps it at 100 (heads.py:95), and the checkpoint's own exp(logit_scale) is 100.81. So the head stack amplifies a cosine error one hundredfold before the softmax: a 0.001 cosine error already costs 0.10 nats, and 0.02 costs 2.0. Mean cosine similarity is therefore close to useless as a quality signal here, and flips concentrate on near-ties. "Confident" means the reference's own top-1 probability exceeded 0.20 — that column is the one to read.

⚠️ Not recommended for ranking: Q4_K_M. planner accuracy falls 6.63 points, beyond the 3.0-point budget. Published with the measured numbers so the result is reproducible, not because it is a drop-in replacement for the bf16 encoder.

ℹ️ Usable, but not recommended: Q6_K, Q5_K_M. Every decisive decision is preserved, but accuracy against the T-Rex planner's own label drops: Q6_K -2.88 points; Q5_K_M -2.05 points. Use these if you need the size; verify on your own distribution first.

Against the numbers published in the parent model card

case	model card (vLLM bf16)	this runtime (bf16 reference)
anchor-invoice/department	billing	billing — argmax matches, billing=0.98775, technical=0.01225
anchor-tides/rank	0	0 — argmax matches, 0=0.99340, 1=0.00003, 2=0.00657

The model card's probabilities were produced by vLLM in bf16. llama.cpp bf16 reproduces the arg-max exactly on both cases and the tides probability to four decimals (0.9934 against the card's 0.993), but the two-option invoice margin comes out more confident than the card's. Tokenisation is not the cause: llama-tokenize and AutoTokenizer produce byte-identical id sequences, with no BOS and no EOS added on either path. It is a bf16 cross-runtime numerics difference, and it is exactly why every comparison in this repo is made against a same-runtime bf16 reference rather than against the vLLM numbers — otherwise a runtime delta would be reported as a quantisation result. The cross-check that the pooling here is right is that two independent implementations of the same encoder (llama.cpp and MLX), both bf16, agree to a cosine of 0.998680 minimum and 0.999958 mean over all 4,448 texts.

A second line: *-outq2.gguf, with output.weight at q2_K

Each of these is the same quant as the file of the same name without the suffix, built from the same BF16 source with the same imatrix, differing in exactly one flag: --output-tensor-type q2_K instead of q8_0. output.weight is the lm_head — 622,329,856 parameters, 7.6 % of the model.

It is safe because it was measured, not assumed. Pooling reads the last hidden state and never a logit, so the output projection is not in the graph. Rebuilding with output.weight at q2_K — a 4x precision change on that tensor — produces bit-identical embeddings: max absolute difference 0.0, cosine exactly 1.0, over the corpus. The same conclusion was reached independently on the MLX side, where zeroing every lm_head tensor rather than re-quantising it also changed nothing. So the saving below is free; there is no accuracy trade-off being made.

file	bytes saved vs its -outq8_0 twin
Qwen3-8B-Q8_0-outq2.gguf	457 MB (8.71 GB -> 8.25 GB)
Qwen3-8B-Q6_K-outq2.gguf	457 MB (7.03 GB -> 6.57 GB)
Qwen3-8B-Q5_K_M-outq2.gguf	457 MB (6.24 GB -> 5.78 GB)
Qwen3-8B-Q4_K_M-outq2.gguf	457 MB (5.49 GB -> 5.03 GB)

⚠️ Note — these files are for pooling only, not for generation. A q2_K lm_head is a deliberately wrecked output projection. It is never evaluated under --pooling last --embedding, so the embeddings and every number in this card are unaffected — but if you load one of these files without pooling, for a completion or a chat, it will emit nonsense logits and you will get garbage. Use the non--outq2 file if you want the same weights in a form that could plausibly be loaded as a causal LM, at 457 MB more. This is a deliberate, measured size optimisation for the ranker use case, not a general-purpose quant.

Measured and rejected: IQ4_XS

type	bpw	top-1	decisive (>1 nat)	planner acc. Δ
IQ4_XS	–	0.6634	0.9189	-27.99 pts

IQ4_XS is an i-quant: 4-bit weights chosen from a shared 16-entry codebook per 256-weight super-block, and in llama.cpp it routinely matches or beats Q5_K_M on perplexity while being smaller than Q4_K_M. On this encoder it is not close. It is shipped by nobody here precisely because it fails, but the measurement is published because "smaller 4-bit i-quant" is a common assumption and it does not survive contact with a trained embedding model: a 4-bit codebook has nowhere to put this model's outlier channels, and an importance matrix helps rounding far more than it helps codebook selection. Q4_K_M remains the best 4-bit option, and it is still not recommended.

Quantization details

Tool: llama.cpp b11200, llama-quantize, from a bf16 GGUF of Qwen/Qwen3-8B. All three runs used the same flags:

--imatrix imatrix.gguf --override-kv pooling_type=int:3 \
--token-embedding-type q8_0 --output-tensor-type q8_0 <bf16.gguf> <out.gguf> <TYPE> 16

• Imatrix over 10,200 sequences / 214,244 tokens drawn from the CLM decision corpus itself — states, candidates and long documents — not generic web text. Run at -c 2048; the tool's own default of 512 is far below CLM's real sequence length. A real CLM state averages ~21 tokens, so the calibration set is many short sequences rather than a few long ones; what the quantizer consumes is the number of token positions accumulated into Σ(act²).
• Last-token pooling is baked in via --override-kv pooling_type=int:3. Without it convert_hf_to_gguf.py writes no pooling KV at all (Qwen3-8B has no modules.json) and the file resolves to pooling type none, which makes /v1/embeddings answer Pooling type 'none' is not OAI compatible with a 400. llama-server prints model default pooling_type is [-1], but [3] was specified when the flag is doing the work.
• No EOS is appended. Qwen3-8B's tokenizer_config.json sets add_bos_token: false, and the heads were trained on add_special_tokens=False text. The <|im_end|> fix Qwen3-Embedding needed would break fidelity here.
• --token-embedding-type q8_0 holds token_embd.weight (622 M params, 7.6 % of the model) at 8-bit. It is the most outlier-prone tensor in the encoder and every pooled vector starts there. The cost is the extra quant bits on 622,329,856 parameters — 311 MB against a 4.5 bpw bulk — not the scale storage.
• --output-tensor-type q8_0 holds lm_head at 8-bit. This is provably neutral, not merely assumed: rebuilding the identical model with output.weight at q2_K instead — a 4x precision change across 622,329,856 parameters — produces bit-identical embeddings (max abs difference 0.0, cosine 1.0). The output projection is not in the pooling graph at all. It is left at 8-bit anyway, at a cost of 457 MB, because a q2_K head would silently emit garbage logits for anyone who tried to use one of these files as a causal LM; cheap insurance against that footgun. Note the contrast with token_embd.weight above, which is very much alive.

Nothing was requantised: all three come from the BF16 GGUF, never from another quant, and --allow-requantize was not used.

Limitations

• The projection heads are encoder-locked. They were trained against Qwen3-8B last-token-pooled 4096-d embeddings, so these files are drop-in only if the pooling and the head checkpoint both match. clm_config.json records both.
• The lm_head inside these files is not evaluated. Measured, not assumed: 4x lower precision on output.weight leaves the embeddings bit-identical.
• This is a ranker, not a generator. lm_head is never evaluated under pooling; these files are not useful for text generation.
• Probabilities are relative to the candidate set. Report rank metrics rather than absolute probabilities, and read the margin: a flip on a near-tie is noise, a flip on a confident decision is a defect.
• Quantisation shifts InfoNCE logits by 100× the cosine error. See the Evaluation section; this is the dominant effect and it is why a good mean cosine can still accompany a bad top-1 rate.
• -c is the total context, split across -np slots. -c 2048 -np 8 gives each slot only 256 tokens, and long states will fail. Use -c 16384 -np 8 for a 2048-token window.
• Upstream asymmetry, reproduced here: CLM's training recipe keeps the last tokens of a state while the serving path truncates from the head. These quants follow the serving path, which is what is actually deployed.

License

Apache-2.0, inherited from Qwen/Qwen3-8B and Contrastive-LM/CLM-v0.1-8B, both Apache-2.0 and ungated. This repo is not gated; gating an Apache-2.0 model would misrepresent its terms and break anonymous from_pretrained.

Citation

@misc{kwok2026contrastivelanguagemodels,
  title={Contrastive Language Models: A System One Model for Fast and Generalizable Decision-Making},
  author={Jacky Kwok and Hangoo Kang and Tarun Suresh and Jon Saad-Falcon and Marco Pavone and Christopher Ré and Azalia Mirhoseini},
  year={2026},
  note={Notion Blog},
  url={https://contrastive-lm.notion.site}
}