colibri v2.1.02026-10-31
Huge open models on the machine you already have: on the CPU, on any GPU through Vulkan, on NVIDIA cards through CUDA. Pure C, one file per model family.
387 pull requests since v2.0.0, 208 of them from 35 contributors. GGUF files now run natively on every engine, read exactly as published. Four new models arrive, Mistral-Large-4 (1T) among them. CUDA gets device chains of its own and a routed-expert tier for every engine. Qwen3.8-Flash-Next runs a 256k context at the speed of 64k, an agent's turns read only what is new, and there is a new chat UI.
<p align="center">
<img src="https://raw.githubusercontent.com/JustVugg/colibri/v2.1.0/docs/media/ui/ui-experts.png" width="900" alt="the new chat UI with the expert map open">
</p>
Highlights
- GGUF on every engine. coli import takes a GGUF where it lies and every engine runs it as published: no conversion, no second quantization, from IQ1S to Q80, F16, BF16 and MXFP4, the same blocks and the same tokens on the CPU, Vulkan and CUDA. unsloth's Dynamic UD quants run as they are (docs/gguf.md).
- New models. Mistral-Large-4 (1T, 52B active: text, pictures, reasoning, tools), Muse Spark Open (816B), Flint (40B, 2.5B active), LFM3-VL-2B with its vision tower, and the published DeepSeek V4.1 Flash and Kimi K3 GGUFs.
- CUDA. Every layer of Qwen3.6, Qwen3.8 and OLMoE as one chain on the card, a routed-expert tier on the card for every engine, the dense matrices on the card for DeepSeek V4, V4.1, Kimi K3, GLM-5.3 and MiMo, and a GGUF's blocks on CUDA with the prompt on the tensor cores.
- Vulkan. A model larger than the card keeps the experts the card cannot hold in a host pool in RAM, and the CPU computes them inside the device's frame while the card computes its own. Prompts on the matrix units, flash attention with windows and sinks, float16 KV mirrors.
- Long contexts and agents. Qwen3.8-Flash-Next at 262,144 tokens: decode 9.6 to 45.7 tok/s, prompt 15.6 to 149. A short request between two turns (a title, a summary) parks the conversation instead of taking its state, on every engine: a Hermes Agent turn went from 86 s to 4.9 s.
- A new chat UI. coli ui: reasoning in its own block, pictures, web search, a live map of the experts as they fire, a monitor, your models, and coding agents already pointed at colibri. Seven languages.
- Speculative decoding. Qwen3.6's MTP head on every backend (Q36_MTP=1): decode 99.7-104.5 to 115.8-120.1 tok/s on CUDA, 83.2-83.8 to 99.9-109.3 on Vulkan; DeepSeek V4.1's DSpark head drafts as deep as its confidence says, behind a gate that skips the drafts that cannot pay.
- Memory. KV caches in float16: GLM-5.3 at 131,072 tokens in 11.9 GB instead of 23.9.
Measured on one 24 GB card
An i5-13500 with 62 GB of RAM and an RTX PRO 4000 Blackwell SFF (24 GB, 70 W), each model served by coli serve: a fresh 450-token prompt, then a 128-token streamed answer, the request's HTTP included; the better of two runs, tok/s.
| GGUF | CUDA prompt | CUDA decode | Vulkan prompt | Vulkan decode | CPU prompt | CPU decode |
|---|---:|---:|---:|---:|---:|---:|
| Qwen3.6-35B-A3B UD-Q4KM | 3,491 | 104.8 | 2,076 | 92.9 | 122.6 | 13.9 |
| LFM3-VL-2B Q4KM | 12,317 | 235.1 | 8,174 | 214.1 | 286.5 | 34.5 |
| OLMoE-1B-7B-0125-Instruct Q4KM | 10,418 | 354.6 | 6,867 | 299.4 | 279.9 | 47.4 |
| LFM3-VL-2B F16 | 11,524 | 103.4 | 7,767 | 99.3 | 49.7 | 6.4 |
| Qwen3.8-Flash-Next IQ2_XS (68 GB: the experts the card cannot hold stay in RAM) | 217.1 | 51.4 | 214.3 | 50.6 | 38.3 | 9.3 |
Bigger than the RAM, on the same machine
Each model from its published GGUF; what neither the card nor the RAM holds comes from two NVMe drives. Tokens per second, from each model's page in docs/.
| GGUF | size | backend | prompt | decode |
|---|---:|---|---:|---:|
| Inkling-Small UD-IQ2_XXS | 77 GB | Vulkan | 15.1-33.1 | 10.92-11.53 |
| MiMo-V2.6-Flash Q2KS | 106 GB | Vulkan | 8.2 | 7.8 |
| DeepSeek V4 Flash 0731 UD-IQ2_XXS | 85 GB | Vulkan | 19.7-20.4 | 6.18-6.25 |
| DeepSeek V4.1 Flash UD-IQ1_S | 233 GB | Vulkan | 31.8 over 3,403 tokens | 5.01-5.29 |
| GLM-5.3-Flash UD-IQ2_XXS | 95 GB | Vulkan | 9.3 | 2.92-3.40 |
| Muse Spark Open UD-IQ1_S | 176 GB | Vulkan | 5.0-5.3 | 1.55-1.58 |
| Mistral-Large-4 UD-IQ1_S | 233 GB | CUDA | 240 tokens in 66-82 s | 1.34-1.36 |
| GLM-5.2 UD-IQ1_S | 202 GB | Vulkan | | 0.633-0.658 |
| Kimi K3 UD-IQ1_S | 594 GB | Vulkan | 31 tokens in 46.5 s | 0.349 |
Run a GGUF
``bash
coli import model.gguf # the first part of a split file, its other parts next to it
coli chat --model model.colibri # or coli serve, then coli ui
`
unsloth's DeepSeek V4.1 Flash, for example:
`bash
hf download unsloth/DeepSeek-V4.1-Flash-1017-GGUF --include "UD-IQ1S/" --local-dir v41
cd v41/UD-IQ1_S
coli import DeepSeek-V4.1-Flash-1017-UD-IQ1_S-00001-of-00005.gguf
COLIVULKAN=1 coli chat --model DeepSeek-V4.1-Flash-1017-UD-IQ1S.colibri
`
Downloads
| file | for |
|---|---|
| colibri-v2.1.0-linux-x86_64.tar.gz | Linux x86-64 (AVX2), every engine with Vulkan built in |
| colibri-v2.1.0-windows-x86_64.zip | Windows x86-64 (AVX2), every engine with Vulkan and the AMD NPU lane; coli.cmd to start |
| colibri-v2.1.0-windows-x86_64-cuda.zip | NVIDIA cards on Windows (compute 8.0 and newer): unpack it over the Windows archive |
| colibri-v2.1.0-macos-arm64.tar.gz | Apple Silicon; colibri, inkling and kimi_k3 with Metal |
| SHA256SUMS.txt | the checksums of the archives |
The one-step setup (START-HERE.bat on Windows, ./start-here.sh on Linux and macOS) picks the archive for your machine, or builds from source when a compiler is there. Every archive was unpacked and checked by CI before publishing: every engine present, coli dispatching to each one, the Vulkan engines finding their shaders, the dashboard and the chat UI loading.
What changes by default
- The KV caches are float16: colibri.c's MLA latent cache (GLM-5.2/5.3, Mistral-Large-4; KV16, not with CUDA or Metal), qwen36's (Q36KV=f32 keeps float32) and qwen38's when serving (Q38KV). Half the RAM of float32.
- A short request between two turns parks the conversation on every engine serving one conversation (COLI_PARK=0` never parks).
Thank you
To the 35 people whose pull requests are in this release: @rudycelekli, @CoTi1938, @bobsummerwill, @Avicennasis, @appleguy, @Stamina9, @kreuzzelg, @simaotwx, @Altroo, @benmaster82, @namespaceMarcello, @JackKnifeAI, @leonwieser2009-oss, @Ontahm, @Varshith-Kali, @djairjr, @neerajdad123-byte, @dajiaohuang, @mfranzon, @huppiflupp, @seanfalling, @jameswitney, @KhalidAlnujaidi, @HyperionBurn, @TyrionV8, @kflirik, @bherald, @Anionex, @mfethe1, @ZacharyZcR, @ggeorgovassilis, @neoo69, @crichalchemist, @enitimeago and @pavolbauer. And to everyone who opened an issue with numbers from their own machine.
Everything in 2.1.0
New models
- #2493, #2500, #2503: Mistral-Large-4, the text model from every
published GGUF and from the FP8 release, on the CPU, the Vulkan chain and CUDA,
and its pictures (the Pixtral tower, from the GGUF's mmproj or our containers).
- #2489, #2491: LFM3-VL's text model (from its GGUFs or the converted
checkpoint) and its pictures.
- #2292, #2297, #2310, #2316: Flint, its safetensors and every
unsloth GGUF, on the CPU, Vulkan and CUDA; its sliding layers keep their K/V in
a ring.
- #2293, #2324: Muse Spark Open from the unsloth GGUFs, CPU, Vulkan and
CUDA.
- #2143, #2269: DeepSeek-V4.1-Flash-1017 as it is, and the published
DeepSeek V4.1 GGUFs with their DSpark head.
- #2274: the published Kimi K3 GGUFs, on the CPU, the Vulkan chain and the
tier.
- #2102: Qwen-Image gets a single-frame CPU VAE encoder.
GGUF on every engine
- #2130: GGUF models read natively, the CPU kernels interleaved by 16 rows.
- #2140, #2141: DeepSee
colibri v2.0.02026-10-06
169 pull requests since v1.12.1, 88 of them from contributors. Every MoE
engine now runs on any GPU a Vulkan driver can see: the routed experts on a
shared device tier, the dense layers on a device chain (all of them, or the
first N that fit), and a second GPU for more experts. Three model families
arrive (MiMo-V2.6, Qwen3-Coder, Qwen-Image-2.1) and one dense checkpoint
(Qwen3.8-27B). Qwen3.8 Flash Next gets int4 experts and speculative decoding
that is on by default. Every text engine serves up to 16 conversations at once.
The release archives carry the GPU backends. Setup is one step, and there is one
decision API (System One).
Several conversations at once
- #1955: every text engine serves several conversations at once (coli serve
--kv-slots N, up to 16), as GLM-5.2 and GLM-5.3 Flash did: qwen36 (Qwen3.6,
Qwen3-Coder, Qwen3.8-27B), qwen38, OLMoE, Inkling, Kimi K3, MiMo-V2.6,
DeepSeek V4 and V4.1. Each slot keeps its own conversation's state, and every
decode step takes the next token of each active request as one batch, so the
weights and experts a step reads serve all of them. A request gets the tokens
it gets alone, checked frame for frame on every engine, on the CPU and with
the expert tier. With more than one slot nothing drafts and the dense chain
stays off.
Vulkan on every engine
- #1830: every engine's resident matrices run on any Vulkan GPU (fmt 10, 11
and 12 added), each configuration checked against the engine's own CPU run on
Lavapipe (tests/vulkan_engines.sh).
- #1834: tiled GEMMs for prefill in every weight format, and a
cooperative-matrix GEMM where the device has one.
- #1840, #1843: a shared routed-expert tier (vk_tier.c). It holds a
cache of experts in device memory, filled from the expert history and adapting
while you chat, and computes the resident experts of each step as one async
batch while the CPU computes the rest. First on qwen38 and qwen36, then on
every MoE engine: GLM-5.2, GLM-5.3 Flash, Inkling, OLMoE, Kimi K3, MiMo-V2.6,
DeepSeek V4 and V4.1.
- #1844, #1848, #1849, #1851, #1857: the dense chain
(vk_chain.c), a layer's dense part in one submission, on qwen36, qwen38,
MiMo-V2.6 (sliding window and sinks on the device), OLMoE, Inkling (and
OLMoE's PILOT race fixed), GLM-5.2 and GLM-5.3 Flash (shared MLA, DSA, KDA and
mHC ops), Kimi K3, DeepSeek V4.1 and DeepSeek V4.
- #1871: staged device-local uploads for cards without Resizable BAR, where
the mapped path failed or spilled to system RAM.
- #1901: expert streaming for big prompt steps, memory budgets and
speculative verification on the device.
- #1909, #1913, #1916, #1918: a partial dense chain. The first N
layers that fit go to the device, the CPU runs the rest, and nothing is
uploaded that the chain will not use, so an 8 GB card with a larger trunk keeps
the chain and its expert tier instead of losing both. COLIVKCHAIN_LAYERS
forces N, coli plan predicts it. DeepSeek V4, qwen36 and olmoe, qwen38,
GLM-5.2 and GLM-5.3 Flash.
- #1920: a second GPU for the expert tier (COLIVKDEV2=auto|<index>) on
every MoE engine: the experts after the primary device's, each step one batch
per device, both in flight at once. A failing second device gives its experts
back to the CPU and the tier goes on with the first.
- #1942: the dense chain's layers on two GPUs. With COLIVKDEV2, the
layers a partial chain leaves go to the second device instead of the CPU, on
every chain engine (qwen36, qwen38, OLMoE, MiMo-V2.6, Inkling, GLM-5.2,
GLM-5.3 Flash, Kimi K3, DeepSeek V4.1 and V4). COLIVKCHAIN_LAYERS2 forces
how many, COLIVKCHAIN_DEV2=0 keeps them off. A lost second device leaves
the CPU to run its layers, the recurrent state rebuilt where it was held.
Checked on Lavapipe opened twice; two real GPUs not measured yet.
- #1933: COLIVKKV_COLD=device (opt-in): with the KV cache split past the
device's budget, the host's part of the attention runs on the device too, from
host memory it reads in place, so a step runs in one frame. The prompt gains
4-6% on a Radeon 780M, decode is slower there (hence off by default); a
dedicated GPU has not been measured.
- #1948: several conversations at once on the dense chain. colibri's
multiplexed decode (KV_SLOTS, one row from each active conversation) runs on
the device: the batch's matrices as one, each row's attention over its own
conversation's KV mirror (up to COLIVKCHAIN_MUX, default 16, beside the
chain's own) with its own DSA list; two devices and the partial chain too.
Every frame checked against the CPU's on Lavapipe; not measured with a real
checkpoint yet.
- #1934, #1935, #1936, #1937, #1938, #1939 (@huppiflupp):
the matrix units for prompt attention and int8/int4 prompt GEMMs on the
chain, a step's experts as one grouped GEMM (decode: one grouped GEMV per
phase), qwen36's routing in parallel, and int8 decode GEMVs sized by the
matrix. On a Radeon 780M with Qwen3.6-35B-A3B the first token of a
1000-token prompt comes in 9.3 s instead of 12.6, decode 6.1 tok/s instead
of 5.7, perplexity unchanged.
- #1954: on an Intel GPU the dense chain decoded garbage (Qwen3.6 on an Iris
Xe, the default there). Its pipelines now run at the subgroup size the device
reports; the device compiles at 8 to 32 lanes, and the chain's GEMVs had run
at another width than the one they read. On Windows the chain also finds its
optional shaders (MLA, KDA, mHC, DeepSeek V4, the split KV cache), and the
split cache frees its host memory with the matching call.
- #1911: COLIVKDEV=<index> picks the device on a machine with two GPUs of
the same kind; the cooperative GEMM's epilogue stride is aligned to 16 bytes, a
validation error on some drivers (#1908).
- #1846 (@crichalchemist): correct expert-batch results under MoltenVK's
default mode.
- #1762, #1763 (@Kenneth-Javier): the Vulkan loader linked on Windows
under -static, and the Vulkan objects linked into every engine-including
test rule.
Models
- #1813, #1819, #1820, #1827, #1835: MiMo-V2.6 (Flash and
Pro): the engine, the vision tower, tool calling and Brio mode with logprobs,
read as released; Pro verified against Xiaomi's reference on the real
checkpoint, and coli serve names the loaded variant.
- #1828, #1829: Qwen3-Coder-30B-A3B (qwen3_moe) on the qwen36 engine.
- #1777, #1779: dense checkpoints of the Qwen3.6 family on the qwen36
engine (Qwen3.8-27B), text and images.
- #1780: Qwen-Image-2.1, text to image: a C engine, pictures in the
terminal, the web app and the OpenAI images API.
- #1951: Qwen-Image on the GPU and a faster start. The weights load in
7.5 s instead of 34 (four layers at once, quantized straight from bf16, the
same int8 bytes). With Vulkan every block of a denoising step runs on the
device, its attention included, and the VAE decodes there too; a card that
does not hold the 7.1 GB transformer keeps the blocks that fit and streams
the others each step. On a Radeon 780M a 512x512 image takes 61 s instead of
112 and a 1024x1024 step 38 s instead of 67; every stage checked against the
diffusers reference.
- #1831: Qwen3.8 Flash Next's routed experts as int4-g64: 1.4 to 1.56x
decode.
- #1832, #1917: speculative decoding with the checkpoint's MTP head,
lossless, +12-14% on the release; with #1917 it is on by default with prompt
lookup on every engine that has it (Q38MTP=0, COLILOOKUP=0 turn them
off), and coli plan counts the MTP head's memory.
- #1814: Inkling reads the embedding norm under the name transformers 5.18
writes.
System One: one decision API
- #1856: POST /v1/systemone, one decision API, and the Laya decision
engine; /v1/brio is removed.
- #1868: GLiNER2.5-Decide as a
colibri 1.12.12026-09-24
96 pull requests since v1.12.0, 80 of them from contributors. Two tokenizers
brought back to the reference, brio on the ninth engine, coli chat working
again at the default context on two families, and a placement decision that
is now measured on the card in front of it instead of predicted.
Tokenizers, measured against the reference
- #1654: qwen36 tokenized differently from HF tokenizers in two ways.
An added token right after punctuation was encoded as text (X.<|im_end|>
was 7 tokens instead of 3, every chat turn ending in punctuation paid +4,
#1653), and a whitespace run followed by a non-space was one piece where
the regex's \s+(?!\S) leaves the last char to the next one, so every
indented line of code tokenized differently. Measured on the real
vocabulary: 2,803 lines and blocks of code, Markdown, Chinese and
Japanese went from 757 identical to 2,803, with 5.4% fewer tokens.
- #1656: OLMoE's tokenizer.json has no Split, a bare ByteLevel with
use_regex, for which HF runs the original GPT-2 pattern; tok.h applied
cl100k. A GPT-2 family in tok.h: 1,560/1,708 identical before, 1,708/1,708
after. The same measurement on GLM-5.2/5.3/5.3-Flash, DeepSeek V4 and V4.1,
Inkling and Qwen3.8 came back identical on every case.
Brio and the serve contract
- #1662: POST /v1/systemone, the request and the reply of TypeSafe's
Jev API, served by the brio channel: a client written for it points at
colibri and changes the base URL. noul is a yes/no question, choice
scores the labels with their descriptions in the text, score the level
numbers with the expected value and the legend; confidence by their
documented formula. Any model name is accepted on that route. Measured on
the real Qwen3.6: the three-question example of the docs in 1m46 with the
state read once.
- #1655: the DeepSeek V4 engine speaks the numeric channel (logprobs=k,
pin=1, max_tokens=0), so /v1/brio works on the ninth engine instead
of answering 500 (#1648). The head that used to keep only its argmax now
returns the whole row; ECHO per prompt position during prefill, the
prompt-end scores kept with a state snapshot on pin, and a logprob tail
on every DATA frame during generation. The tiny fixture pins that the
best ECHO token equals the greedy token from the same prefix, and that
the pinned predictor equals a cold prefill's.
- #1659: qwen36 and qwen38 refused a request when prompt + max_tokens
exceeded the context, and the gateway's default budget for these two
families is 8192, the whole default context: every request without
max_tokens and every coli chat message answered 400 on a two-token
prompt (#1641). max_tokens is now a ceiling, clamped to the room the
prompt leaves, as GLM and DeepSeek V4 already did; only a prompt that does
not fit is refused. docs/api.md states the rule, docs/qwen38.md names
Q38_MAXT as the variable --ctx becomes.
The dense trunk in VRAM, measured before it is placed
- #1657: qwen36 offers the rest of its dense trunk to the VRAM placer:
the DeltaNet out_proj (dnout), the attention q/k/v/o (attnproj) and
the shared expert (shexp), about 650 MB more of int8 on the 35B beside
lmhead and dnproj. Measured on four Tesla M10 by the reporter of
#1652, every placed component ran slower than the CPU (lm_head 68.8 ms
against 41.7), so the engine now times one GEMV both ways at startup and
withdraws the whole automatic placement when the GPU loses, giving the
VRAM back to the experts: auto equals off on that box, byte-identical
output. A hand-written COLIPLACE stands; COLITRUNK_PROBE=0 trusts
the placer.
Performance
- #1664: qwen36's dense trunk and routed experts multiply with integer
dot products. The activation is quantized to int8 once per call and the
weights, int8 rows or int4 planar blocks, meet it with maddubs / vpdpbusd
instead of a float conversion per weight; the integer kernels move from
quant.h into idot.h, shared by every engine. Measured on the 35B, 8
threads, every expert resident: decode 6.71 to 8.23 tok/s (+22.6%), lm_head
12.6 to 10.1 ms/token, the expert compute 22.7 to 15.6, for +1.3%
perplexity on 4 x 512 tokens. Both are the default (COLIDENSEIDOT=0,
QWENEXPERTACT=f32 restore the f32 kernels). COLIDENSEBITS=4 with
COLIDENSEINT4=<components> stores part of the trunk as int4 in blocks
of 64: opt-in, with the perplexity it costs per component in the docs
(lm_head alone +2.4%, everything +10%).
- #1668: qwen38's dense trunk (553 matrices, 3.6 G weights, 8 GiB of
BF16 read on every token, more than the ten routed experts) is kept on the
CPU as int8 rows with the BF16 copy released, and multiplied with the same
integer kernels; the routed experts' e4m3 blocks are decoded eight at a
time in registers and multiplied with FMA instead of one table lookup per
weight. Measured on the released Qwen3.8-Flash-Next-FP8, 8 threads, RAM
LRU 96 per layer: decode 0.61 to 1.42 tok/s, the trunk 434 to 85 ms/token,
lm_head 76 to 13, the expert GEMVs 388 to 140, peak RSS 32.2 to 28.5 GB;
prefill of 512 tokens 495 to 149 s. Perplexity on 4 x 512 tokens +0.5%
(two chunks lower, two higher); the vector FP8 kernel alone reproduces the
BF16 run to four decimals. Both are the default (Q38TRUNKCPU_INT8=0
keeps the BF16 trunk, Q38FP8KERNEL=scalar the table kernel).
Performance, from contributors
- #1606: the K1b grouped int4 family gets a multi-row tile and AVX-512
and AMX arms, and is no longer switched off on AVX-512 builds; exact on
all eight engines on a 16-core AVX-512 host.
- #1239: an SSE4.1 tier for the olmoe and qwen36 int8 GEMV, for hosts
with SSE4.1 but no AVX2; on the Sandy Bridge of #1652 decode went from
2.45 to 3.54 tok/s.
- #1313: matmul_fp8 computes four output rows per pass under clang,
where the contraction makes it bit-exact; GCC keeps the one-row kernel.
- #1612: qwen36 gains the GLM engine's CACHE_ROUTE lever, with the
VRAM tier as the first residency level, opt-in.
- #906: DEGRADE_ZERO, an opt-in policy that zero-fills a missed
expert slot below a gate-weight threshold instead of blocking on the
load (#865).
- #1677: qwen36 projects a prompt's DeltaNet inputs (qkv and z) on the
card in blocks of up to 256 rows instead of one row at a time; a 259-row
prefill makes 2 projection calls instead of 259, with the convolution
history and the recurrent state checked against the CPU run. The paired
microbenchmark on an RTX 4070 read 4 to 10x per projection.
- #1674: qwen36's attention projections the tier placed in VRAM answer
a whole prompt batch with one call per matrix; a failed call turns only
that handle off and the prompt continues on the CPU.
- #1676: Kimi K3's streaming CUDA expert keeps the gate, up and SiTU
intermediates on the device and applies down there (one fused entry
point, optional in the DLL: an older backend keeps the three-call path).
- #1673: the streaming MXFP4 matmul reuses one grow-only device scratch
per card instead of allocating and freeing weights and scales on every
call.
- #1559 (kreuzzelg): convert_qwen36.py --down-bits 8 writes the mixed
expert layout, int4 gs64 gate/up and int8 down in one slab (5.7 bits per
weight against gs64's 4.5); the engine tells it apart by size and reads
each matrix in its own format on the CPU path, and refuses the VRAM tier
with a line. It is the knob behind the #1370 numbers: on wikitext-2 the
int8 down alone recovers a quarter of the gap between gs64 and all-int8,
the rest sits in gate/up. A measurement tool and a middle step, not the
answer to the gap.
- #1730 (mfethe1): DeepSeek V4's FP4 expert kernels, the prefill batch
and the decode matvec, get a NEON arm; arm64 used to take the scalar
arm, which is why Apple Silic
colibri 1.12.02026-09-20
colibri 1.12.0
81 pull requests since v1.11.0. A new way to ask a model a closed question,
a redesigned dashboard and landing page, and a long run of small failures
that used to answer 500 or die on a locale.
Brio mode: score a closed set instead of generating
- #1632: brio mode, on all nine engines. Hand the engine the options an
answer is allowed to take and it reports the probability of each one
instead of writing the answer: the option tokens are read, not sampled,
so completion_tokens is 0 and no reply can fall outside the list. It is
opt-in per request through two SUBMIT keys, logprobs=k and pin=1;
without them every frame is byte-identical to before, which is asserted
per engine rather than claimed. max_tokens=0 became legal, but only
together with logprobs>0.
- The shared contract is three headers: decode_batch.h for the logprob
tail, servecodec.h for the wire, and pinpool.h for nested state
snapshots, so a request can keep one photograph of the shared
instructions and a deeper one of the instructions plus the question. With
one level the question is re-read once per option; with two, four items
cost 176 tokens instead of 496.
- kvprefixholds() is the guard that makes the snapshots safe: a
snapshot is reused only when the prefix record still holds the same ids,
because a K/V bank can be dropped between two options.
- Exactness is measured, not asserted: the logprobs read from a snapshot
match a cold recomputation to 0.00e+00 on all nine tiny fixtures, and on
the two real checkpoints available (qwen36 at 22 GB and DeepSeek V4.1
Flash at 476 GB).
- Three clients: POST /v1/brio on the gateway, /brio in coli chat,
and a page in the dashboard. docs/brio.md has the request, the reply,
the normalisation choice and the cases where the mode does not help.
- The endpoint takes three forms. options is one closed question.
questions is many questions on one state, each with its own options,
with the state photographed once and the snapshots ordered by the server.
schema is an object of field to allowed values: the server writes the
JSON skeleton and fills it one field at a time, so the object is valid by
construction and every value is one the caller allowed, each with its own
probability and entropy. Both were measured before they were exposed: the
5.7x and the 2.4x above are these two forms.
The dashboard and the landing page
- #1633: the web dashboard is redesigned around a workspace and a
navigation dock, with a brio page that reads a document once and asks it
several questions, each with its own option set, and a light and dark
theme.
- #1635: the landing page lists the models as a searchable list instead
of a card grid, gains a brio section, and gets a light and dark theme
built on the mechanism proposed in #1551. The shipping version and the
family names are generated from c/version.py and c/family_registry.py,
so the page cannot claim eight families again while nine are running.
The hero plays a recorded session rather than a mock-up.
Performance
- #1477: one routed-expert kernel, expert_ffn.h, shared by the MoE
engines and used by qwen36 first: int4 kept planar in RAM instead of
unpacked to int8, and two OpenMP regions per layer instead of three per
expert. 12.8 to 15.7 tok/s with the resident set down from 29 to 17 GB,
output identical.
- #1424: a CUDA VRAM tier for Qwen3.8-Flash-Next, 43 % on one 8 GB card
and 60 % on two.
- #1553 and #1558: the previous turn's KV is reused instead of
re-prefilling the transcript, on qwen36, olmoe and DeepSeek V4.1, with a
resumed prefill that is exact.
- #1521 and #1524: the mHC mix and the KDA convolution and head
loops run in parallel, bit-identical, for 3.9 % and 3.0 %.
- #1517: the four engines that never sized their OpenMP team now do.
- #1522: the DeepSeek V4.1 expert stream can be read from more than one
drive; #1455 weights glm53 expert reads by disk and honours
COLIMODELMIRROR; #1543 fixes the V4 mirror cache on unified memory.
- #1495: less overhead in the sparse router's top-K selection.
Fixed
- Requests that answered 500 engine failed and now answer properly:
a non-object jsonschema (#1587**), an unpaired surrogate escape
(#1589), a non-object toolchoice.function (#1598**), and a tool
call whose arguments are not an object.
- #1595: the grammar draft walker switched itself off after about sixty
repetitions. #1596: a nested array schema doubled the compiled
grammar at every level.
- #1483 and #1514: olmoe, inkling and glm53 refuse an over-long
prompt with CONTEXT_EXCEEDED instead of failing late.
- #1549: the lazy shard-mapping table in st.h was published without
synchronisation and glm53 reached it from an OpenMP region.
- #1557: a truncated multibyte tail made the qwen36 tokenizer read past
the prompt. #1530: an integer overflow check in kimi_k3.c.
- #1631: coli serve reaches the expert-history save when it is stopped
with SIGTERM, not only on Ctrl-C. #1628: the expert grid is resent
after every turn, so the dashboard stops showing the cold snapshot
forever.
- #1608: coli doctor matches core tensor roles by component instead of
by one family's spelling.
- #1526, #1584, #1585, #1536: the planner, the auto-tuner
and the registry agree about which engine is running, and stop advising
GLM-only knobs on every family.
- #1489: coli bench ran the GLM engine on any model. #1503: every
engine is handed its model directory through SNAP. #1486 and
#1488: a directory without config.json names no engine, and
coli convert refuses an output directory that already holds one.
Windows, macOS and locales
- #1515, #1560, #1561, #1562, #1516: the chat, coli
run, coli tune and the datapoint harness no longer die when an engine
line, a prompt or a dash does not fit the console code page.
- #1537, #1579, #1535: the qwen36 CUDA_DLL build on Windows gets
its tier, is recognised, and the device probe loads the backend before
counting devices. #1563: the Inkling dense int4 converter finishes.
- #1599: MacPorts libomp built every macOS engine single-threaded.
#1548: Metal falls back to MTLCopyAllDevices when there is no
system default device. #1504: olmoe measures available RAM on macOS
and Windows too.
Diagnostics
- #1494, #1506, #1512: PROF reports block time, head time,
matmul, attention and lmhead, at microsecond resolution, on deepseekv4,
olmoe and qwen36. #1555: CONSIST=1 checks prefill against decode.
- #1482: configurable generation statistics in coli chat, with exact
token throughput when the server reports it.
- #1468, #1487, #1480: a failure in the MoE step, a missing core
tensor and duplicate tensor names are diagnosed by name instead of by a
generic message.
Docs and licence
- #1540 and #1541: the copyright holder is named and a NOTICE file
is added.
- #1294: a reproducible benchmarking protocol; #1508: a GPU-backend
datapoint needs a correctness line to be accepted.
- #1471 and #1473: the expired Discord invite is replaced in all
four READMEs and on the site.
Builds for Linux x8664, macOS arm64 and Windows x8664 are attached below by the release workflow, with SHA256SUMS.txt.
colibri v1.11.02026-09-13
v1.11.0 加入第 9 个引擎 DeepSeek V4.1 Flash(552B,磁盘占用 510 GB):官方 checkpoint 直接读取、无需转换,fp8 稠密权重配 32x32 ue8m0 分块,fp4 专家布局与 Kimi K3 引擎已支持的 mxfp4 逐字节一致。架构里非 V4 的部分全部到位——Engram 双 n-gram 记忆(3.84 亿行、
colibri 1.10.22026-09-06
Patch release. Three of these fixes answer reports made against 1.10.1 in the days after it shipped. No new engine; one behaviour change, opt-in.
If you hit one of these on 1.10.1, this is the fix
coli convert produced a GLM-5.3-Flash container the engine refused (#1368). It ran GLM-5.2's converter on everything; Flash nests its language model under the vision wrapper, so the embedding matched no rule, fell through to the generic fallback, and was quantized. The engine was right to refuse it, hours later, inside coli web. coli convert now reads the checkpoint's config.json and picks the converter from it. GLM-5.2's path is unchanged. An option the target converter does not take is refused, not silently dropped (#1369). And tools/convert_glm53.py, the right tool for Flash, is in the archive for the first time (#1364).
coli doctor said two core tensors were missing from a healthy GLM-5.3-Flash (#1365). The check held three literal tensor names taken from GLM-5.2. It now matches roles, prefix-agnostically, so it holds for any family that nests, including ones not written yet (#1366). Nothing was ever missing from those downloads.
The archive was missing files coli reaches (#1359): iq3_pack.py and its grid, reachable only through a subprocess-launched script. The packer follows that edge now, and data files next to a module (#1364).
Also fixed
- RSS guard counts the memory the process owns, not page cache the kernel can reclaim, so a mapped container no longer evicts experts to free memory it was not using (#1350)
- serve honors CANCEL mid-turn (#1336); the disconnect scenario is built, not hoped for (#1329)
- Metal: bit-exact fp8-e4m3 decode (#1346)
- Qwen3.6: tokenizer merges in both spellings (#1319)
- macOS: Homebrew prefixes found when brew is off the PATH (#1320); DeepSeek V4 reports real CPU and memory on macOS (#1308)
- doctor omits the GPU plan for a CPU-only engine (#1322); a missing core tensor explains itself on every fatal path (#1318)
- GLM-5.3-Flash --no-think is the template's lowest effort level, not a prompt shape of ours (#1327)
- The converter refuses a checkpoint it cannot serve, with a per-family pointer (#1305)
- GLM-5.3-Flash serve can use the 16 KV slots the engine has; the registry declared 1 (#1283)
- Security: imageurl local reads refuse .., and with COLIIMAGE_ROOT set the path must resolve inside it, so an authenticated client of a non-loopback server cannot read arbitrary files through the image API; errors no longer confirm a path exists (#1354, dmoraesrs). CI runs with contents: read and third-party actions are pinned by SHA
Supported checkpoints, stated plainly
- GLM-5.3 (744B-class) runs on the colibri engine. It is the same base model as GLM-5.2 (Z.ai say so; the configs differ only by the transformers version that wrote them), so the family is now named GLM-5.2/5.3 rather than announcing one as the other (#1367).
- DeepSeek V4 Flash REAP-150B (puwaer/DeepSeek-V4-Flash-0731-reap-150b, 85 GB, 132 of 256 experts) loads with the same engine and no conversion (#1310), confirmed by a second user. It is documented, and the banner reports it by its measured geometry, 43L x 132E, not as 284B (#1371).
Changed
- qwen36 VRAM tier promotes int8 experts instead of reserving for nothing (#1334)
- ...and the three tier bugs that surfaced with it are fixed: an is_x overrun with two or more GPUs, a shutdown that could hang, and a use-after-free on int8 containers (#1339, #1340, #1341; fix #1344 by crichalchemist)
- Kimi K3 stops loading a DSA indexer nothing reads (#1335)
- Opt-in COLIMAPEXPERTS=1: experts served through a per-shard file mapping (#1325). Off by default; output byte-identical either way
Docs
- Windows DeepSeek V4 users are led to the release launcher, coli.cmd, instead of a source build (#1291)
Build
- Makefile lists the headers each engine includes as prerequisites (#1284, #1349)
- Site and READMEs carry the eight families with real RAM figures under a contract test (#1302)
colibri v1.10.12026-08-31
v1.10.1 只是打包修复,引擎本身没动:从源码编译的话,它和 v1.10.0 完全一样。
v1.10.0 的预编译包漏了七个 Python 文件,导致 coli convert、coli eval、coli mirror 和 benchmark 拉取全部报「No such file or directory」;经服务端发给 Qwen3.8 的图片也会谎称缺少 Pillow 和 numpy,实际是该带的文件没带上。这次改掉了发布流程里手工罗列文件的做法:打包脚本会顺着 import 和子进程调用算出 coli 真正会碰到的全部文件,发布前再对解压后的产物重算一遍做校验——旧检查只确认拷贝步骤刚写入的那一个文件,所以 v1.10.0 才带着绿灯发了出去。现在每个归档含 14 个 Python 文件,且以实际打开压缩包为准核验。
顺带两处改进:引擎不再接受只读得懂一半的数字参数,./qwen38 3x2 会直接报错,而不是按缓存 3(本应 32)静默跑完还显得很慢;CI 的 Python 任务会先构建测试所需的引擎,七个此前被悄悄跳过的测试恢复运行。
升级只需用新包替换解压目录,模型、容器和参数都不受影响。
colibri v1.10.02026-08-31
Qwen3.8-Flash-Next 是第八个家族,也是第一个整套到货的:引擎、工具调用、视觉一次配齐,而非后续补上。模型为 125B 主干加 51B 哈希 n-gram,48 层(12 组,每组 3 个 Gated DeltaNet 加 1 个 QSA),512 专家取 top-10 外加一个共享专家;常驻 9.2 GiB,cap 32 时合计 16.5 GiB。
``
SNAP=/path/to/Qwen3.8-Flash-Next-FP8 ./qwen38 32
./coli chat --model /path/to/Qwen3.8-Flash-Next-FP8
``
注意本家族不做转换:没有 int4 容器,也没有转换器,须下载官方 FP8 检查点(185.5 GB)直接运行——引擎原生读 FP8,加载时把专家量化进 RAM 缓
colibrì v1.9.02026-08-28
colibrì v1.9.0 接入第七个模型家族 GLM-5.3-Flash:321B 参数、45 层加 MTP,带视觉塔,专家分片从磁盘流式读取,25 GB 内存即可运行,还能读图。MLA 被吸收进缓存——潜向量替代展开后的键,每 token 从 1.39 MB 降到 33 KB,运算完全等价。实测 4-bit 常驻约 12 GB,prefill 工作区恒定 60 MB;但解码卡在磁盘上:QD1 下仅 72 MB/s,每 token 至少 24 秒,换更快的芯片也推不动,只有减少字节数才有意义。图片可贴路径、拖附件或走 OpenAI 的 image_url,附件失败会直接拒绝该轮而非静默发出;预处理与官方处理器对齐,工具调用逐字节校验,--no-think、--effort 可调推理强度。转换脚本逐分片处理,峰值磁盘从 328 GB 降到输出加单个 5 GB 分片,62 个分片共 194.7 GB。另有统一的分布式推理 ABI(Segment 与 Edge)、Windows 安装包及 CUDA 投机验证修复。
colibri v1.8.02026-08-24
v1.7.1 合并了 60 个 PR,核心是三件事:专家查找从扫描降为单次探测、prefill 不再重复读取已有字节、DeepSeek V4 的 CUDA 门槛从 Ampere 下探到 2016 年的显卡。各家族专家命中现在都是 O(1)——DeepSeek V4、Inkling、Kimi K3、GLM、OLMoE、Qwen3.6 一律 1 次探测,旧版最坏要 44 到 219 次,且每次查找都会复核槽位身份,过期记录只会落空、不会取错权重。V4 新增 CUDAARCH=portable-pre-ampere NOTC=1,支持 Pascal 与 Turing;DSV4HYBRID=1 与 COLICUDAMOEDOUBLE=1 为实验项,不设环境变量时输出逐字节不变。Kimi K3 加入 KDA/MLA 路径的 token 级对拍、可选的循环状态检查点 COLIK3CKPT,以及按 Moonshot 渲染器实现的工具调用。KV cache 量化落地 CPU 侧,支持 KV8 与 KV_TQ,flash 与 gather 分支同步适配。coli tune 现在能测量所有引擎,并改用实机测得的内存与缓存上限。风险改动全部 opt-in,所有默认路径都有逐 token 校验兜底。