Instructions to use CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF:TQ1_0
# Run inference directly in the terminal:
llama cli -hf CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF:TQ1_0

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF:TQ1_0
# Run inference directly in the terminal:
llama cli -hf CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF:TQ1_0

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF:TQ1_0
# Run inference directly in the terminal:
./llama-cli -hf CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF:TQ1_0

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF:TQ1_0
# Run inference directly in the terminal:
./build/bin/llama-cli -hf CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF:TQ1_0

Use Docker

docker model run hf.co/CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF:TQ1_0

LM Studio
Jan

vLLM

How to use CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF:TQ1_0

Ollama
How to use CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF with Ollama:
```
ollama run hf.co/CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF:TQ1_0
```
Unsloth Desktop

How to use CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF:TQ1_0

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF:TQ1_0"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF with Docker Model Runner:
```
docker model run hf.co/CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF:TQ1_0
```

Lemonade

How to use CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF:TQ1_0

Run and chat with the model

lemonade run user.Ternary-Bonsai-2-27B-262k-GGUF-TQ1_0

List all available models

lemonade list

Hermes Agent

How to use CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF:TQ1_0

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF:TQ1_0

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF:TQ1_0

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF:TQ1_0" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Bonsai 2 27B, served: the full 262,144-token window at q8_0 on a 12 GB card

A 27B model with its whole trained context window, at q8_0 KV precision, on one RTX 4070 12 GB. 78 tok/s of new text at 32k tokens of context (175 on code), 69 at 64k, 61 at 112k, and the window keeps going to 262k. Lossless speculative decoding at every depth. A server that answers the requests agents and apps actually send. One command on Windows.

This is PrismML's Ternary Bonsai 2 27B (1.58-bit ternary PTQ1_0, 5.9 GB, Apache 2.0), the same weights, nothing re-quantized, with its MTP draft head grafted back on (byte-identical trunk, proof below) and the serving stack from github.com/professorpalmer/bonsai-ada-surgery: a tiered KV cache that puts the first ~119k positions in VRAM and the rest in pinned system RAM with bit-identical output, MTP drafting that pays at every depth, harness-proofing for the apps that send effort: "high" or tiny output caps, and an optional server-side layer (exact API cards, an API check, a sandboxed Python tool) that lifts the model's coding and agentic scores without touching the weights.

RTX 4070 12 GB, served, one slot	decode, new text (tok/s)	decode, code (tok/s)	prefill (tok/s)
4k tokens of context	102	102	1,173
16k	83	242	1,137
32k	78	175	1,031
64k	69	109	867
112k (in VRAM)	61	91	698
131k	30	63	649
180k	17.9	28	547
258k (the window's end)	8.6	12.1	436

Measured 2026-10-08 (bench/depth_table.sh), a fresh prompt at each depth: 400-token greedy answers with the default drafting (lookup + MTP head); new text = an essay, code = a Python module; prefill = cumulative for the whole prompt; display on the CPU's iGPU. Every number has a receipt in the repository (docs/Q8_FULL_CONTEXT.md, docs/RECEIPTS.md, docs/QUALITY.md).

What the serve changes, same weights

	before: the published 12 GB recipes	this serve
window on 12 GB	96k (q4_0 KV to reach 262k, much of it in system RAM)	262,144, ~99 % in VRAM (q4_0)
decode at 245k on 12 GB (RTX 4070)	15.4 tok/s (q8_0, window past VRAM)	60.4 tok/s
decode at 32k / 64k	47.6 / 36.9	78 / 69 on new text, 175 / 109 on code (2026-10-08 table)
decode when the answer copies the context (file rewrite), 4k / 130k / 250k	117 / 63 / 18.5 (MTP only)	352 / 164 / 66 (lookup drafting)
speculative decoding	up to 24k, then off	at every depth, outputs identical to drafting off
apps that send `effort: "high"` (Cline, Kilo, Open WebUI)	HTTP 500 on every request	answered (normalized to medium)
apps with a 256-4096 token output cap, thinking on	cut off mid-think	answered (cap raised to the thinking budget)
HumanEval 164 from an app that sends `effort: "high"`	0 (HTTP 500)	160
tool calls parsed (9 requests)	1 of 9 (JSON asked for in content)	9 of 9 (native XML, server grammar)
AppWorld, all 168 test tasks, ReAct code agent	64.3% (published, stock serve); 16 wrong-format replies, 15 step-cap exits	70.8% (95% CI 63.6-77.2); 1 and 1
long exact-work suite, 37 tasks, same seeds	17 raw	28 behind the layer (13 rescues, 2 losses); 29 on a second seed set
AIME 2025 (60) / MMLU-Pro (100), raw -> layer	52 / 71	56 / 76
HumanEval 164, medium reasoning, sandbox-scored		161

Most of the quality gap people report on Bonsai 2 for code and agents is runtime, not compression: the effort word the template rejects, the output cap that ends a thinking model mid-thought, the tool-call format. The server fixes those; the layer adds exact references and a sandbox. The weights are untouched.

Files

file	what	size
`Ternary-Bonsai-2-27B-PTQ1_0-mtp-procreations.gguf`	PrismML's PTQ1_0 file with the on-policy Q8_0 MTP draft head from ProCreations/Ternary-Bonsai-2-27B-MTP grafted on as `blk.64.*`; every original tensor byte-identical to PrismML's file (proof below). This is the file the launcher uses: lossless speculative decoding, 70.6% draft acceptance, +50-100% decode	6.40 GB
`bonsai-bundle-win-x64.zip`	Windows binaries: the patched llama.cpp (PrismML fork + 66 patches), sm_75 / 86 / 89 machine code (RTX 20 / 30 / 40) plus compute_89 PTX for RTX 50, CUDA 13 runtime included, NVIDIA driver only. Launcher, layer, suite and docs at the release tag (`bundle-20261016`)	617 MB
`start-server.ps1`	the launcher (also inside the bundle): reads free VRAM, sizes the VRAM line, starts the layer and the server

The original Ternary-Bonsai-2-27B-PTQ1_0.gguf without the head also works with everything here (BONSAI_SPEC=0 for no drafting, or let the launcher fall back); get it from prism-ml/Ternary-Bonsai-2-27B-gguf.

Run it (Windows, NVIDIA RTX 20 / 30 / 40 / 50)

git clone https://github.com/professorpalmer/bonsai-ada-surgery && cd bonsai-ada-surgery
hf download CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF bonsai-bundle-win-x64.zip --local-dir .
Expand-Archive bonsai-bundle-win-x64.zip -DestinationPath . -Force
hf download CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF Ternary-Bonsai-2-27B-PTQ1_0-mtp-procreations.gguf --local-dir models
.\start-server.ps1

OpenAI-compatible API on http://<host>:8080/v1, bearer key in artifacts\api_key.txt (created on first run), llama.cpp's chat UI on the same port. The launcher prints the VRAM line it chose. Optional, once: layer\fetch_runtime.ps1 downloads the sandbox runtime for the layer (CPython 3.12 on WASI, checksummed, 14 isolation canaries); without it the plain server runs.

Linux: bash build/build_linux.sh in the repository builds the pinned PrismML source with all 66 patches applied; the full-window command line is in the repository README. No prebuilt Linux binary yet.

Cards: 12 GB is the measured recipe: q4_0 K/V by default from bundle-20261016, 260,352 of 262,144 positions in VRAM with the display on the iGPU, 242,176 with the display on the card (copies, needles and HumanEval at 245k equal to q8_0; BONSAI_CTK=q8_0 for the previous cache). 16 GB and up: the whole q8_0 window fits, the tier switches itself off. 8 GB: the launcher detects the card and uses its 8 GB preset (q3_0 K/V since bundle-20261015b, q4_0 before; a 262k window with 24 GB of RAM, else 131k; a Q4_0 MTP head; from bundle-20261016b prefill attention packs 2 query heads per column group on Turing: cold 245k 1.147x on an RTX 2060 SUPER, perplexity +0.02 %; from bundle-20261017 saved conversations (BONSAI_SLOT_SAVE_PATH, automatic with the layer) restore a 199k conversation ~6 s after a restart on this card; from bundle-20261017b the launcher warns with a spill line when another program fills the card after the start, which on this card halves reads or worse). Measured on an RTX 2060 SUPER that also draws the desktop: 51.2 / 43.1 tok/s at 4k / 32k (q4_0); with q3_0, 80,640 positions in VRAM, 64k 41.0, 128k 22.8, 192k 12.3 tok/s, HumanEval medium 163 of 164 (repository docs/8GB.md, docs/research/release-20261014b/).

For agents and apps

Send tools and let the server format and parse the calls (native Qwen3-Coder XML, grammar-constrained).
Tool-heavy agents: chat_template_kwargs: {"enable_thinking": false} per request. Chat and coding answers: the server default (medium reasoning, 20k thinking budget, forced close).
Clients that cap max_tokens low are fine: with thinking on the server raises the cap.
effort: "high", "xhigh", "low": mapped to medium, answered, logged. BONSAI_HARNESS_PROOF=0 passes words through unchanged.

Knobs

BONSAI_CTX (262144), BONSAI_CTK (q8_0), BONSAI_TIER (1), BONSAI_KV_VRAM_CELLS (auto), BONSAI_VRAM_MARGIN (1000 headless / 1300 with the display on the card), BONSAI_SPEC / BONSAI_SPEC_DEEP (2 / 4), BONSAI_DRAFT_WINDOW (16384), BONSAI_EFFORT (medium), BONSAI_THINK (1), BONSAI_THINK_BUDGET (20480), BONSAI_HARNESS_PROOF (1), BONSAI_LAYER (1), BONSAI_PORT, BONSAI_MODEL, BONSAI_MMPROJ / BONSAI_MMPROJ_GPU (vision projector; image encoder on the CPU by default, no VRAM cost). Full table and the VRAM arithmetic in the repository README.

How it works, in one paragraph each

Tiered KV cache (--kv-vram-cells). The first N positions of every layer's K/V live in VRAM, the rest in pinned system RAM mapped into the same CUDA address range. Kernels are unchanged and output is bit-identical to an all-VRAM cache. Past the line, attention copies the used RAM rows into VRAM with the copy engine first, so decode there is bound by PCIe (4.0 x16 here). The box needs ~8 GB of free RAM for the tail at 262k.

MTP drafting at every depth (--spec-draft-window). The draft head only needs recent context, so its cache keeps the last 16k rows and stays small. With quantized-KV decode on the tensor-core attention kernel (one K/V read per verify batch), drafting pays at every depth; greedy output equals greedy output without the head.

Harness-proofing (--reasoning-effort-allow, --reasoning-max-tokens-floor). The same weights score 0 to 160 of 164 on HumanEval depending on what the client sends. The server absorbs both causes.

Lookup drafting (patch 0039, on by default). An n-gram drafter drafts text that is already in the context, up to 32 tokens at a time, in front of the MTP head. File rewrites 117 -> 352 tok/s at 4k and 18.5 -> 66 at 250k; new text unchanged; same output.

Prefill past the VRAM line (patch 0041). The K/V rows a prefill batch writes past the line go to VRAM first and then to system RAM in whole blocks; the next layer's RAM rows are copied while the layers before it compute. A 258k prompt: 369 -> 436 tok/s, same output.

Many tools cost no decode speed (patch 0042). With speculative decoding the server copies the sampler on each draft step; with tools, the copy of the tool-call grammar searched every grammar element, about 15 ms per step with 107 tool schemas. Now a binary search: 107 tools at 63k, 41.8 -> 61.2 tok/s (2 tools: 63.5), same output.

The layer. A small proxy the launcher starts in front of llama-server on the same port. Exact API cards for the Python modules a request involves (generated from the sandbox's own runtime), an API check that flags names that do not exist, and a sandboxed Python tool with the user's text as input.txt. Clients change nothing.

Proof the weights are untouched

Checked tensor by tensor against PrismML's published file (bench/gguf_tensor_identity.py in the repository, run 2026-10-06): all 851 tensors of Ternary-Bonsai-2-27B-PTQ1_0.gguf are byte-identical in this file; the only tensors added are the 15 of blk.64.*, the draft head. Metadata differs by five added keys (graft.*, qwen35.nextn_predict_layers, block count 64 -> 65) and by the embedded chat template, which is this repository's template: PrismML's, plus a system block of operating rules for tool use (save long command output to a file and read the part you need, and so on) that every measurement here was taken with. The launcher uses the GGUF's embedded template with --jinja, so this file serves with that template; PrismML's original file serves with PrismML's.

file	sha256
`Ternary-Bonsai-2-27B-PTQ1_0-mtp-procreations.gguf`	`5f212d02ff57cb8eaad260fd7ff57bfaff87ae2bc9183a21dd2f127c27252505`
PrismML's `Ternary-Bonsai-2-27B-PTQ1_0.gguf` (the base)	`53107f530aa52eb00912263ab1ee29bd199261c87cd7b4ad4ca1318c1fe33ee3`
`bonsai-bundle-win-x64.zip`	`6285cd5d6ef72e9e13ffadefb20b1ef91dd90afe80e53f3375bae268454e79ce`

Every kernel in the stack is checked against the CPU reference, and the served output is checked greedy token-for-token against the unpatched fork before any speed number is recorded.

Credits and licenses

PrismML for Bonsai 2 27B and the llama.cpp fork this runtime is built on. The serving patches are submitted there (#319-#323, #295, #296, #333; earlier work merged in #214, #216, #221).
sudoingX for the planar-transposed activation layout, the batch-invariant mode, the MTP graft tools and the Hadamard fix for the draft graph (bonsai2-small-gpu, PrismML #217, #218).
ProCreations for the on-policy MTP draft head (Ternary-Bonsai-2-27B-MTP).
Alibaba Qwen for Qwen3.5-27B, the base model.
Testers who filed issues with full logs: @Milor123 (empty JSON-schema unions, the vision projector and GPU sharing, deep-context agent runs) and @outbackdingo (GCC 16 build).
The serving stack, launcher, layer, suite and docs: MIT, Cary Palmer. The weights are Apache 2.0 as published by PrismML; this repository redistributes them with the head added and the notices kept. Not affiliated with, endorsed by or sponsored by PrismML, sudoingX, ProCreations or Alibaba.

Issues and measurements: github.com/professorpalmer/bonsai-ada-surgery.

Downloads last month: 11,326

GGUF

Model size

27B params

Architecture

qwen35

Hardware compatibility

1-bit

Model tree for CaryPalmer/Ternary-Bonsai-2-27B-262k-GGUF

Base model

Qwen/Qwen3.8-27B

Quantized

prism-ml/Ternary-Bonsai-2-27B-gguf

Quantized

(38)

this model