v0.2.02026-10-21
Highlights
- GGUF Support. llama.cpp GGUF files serve directly with ft serve --model <local file>.gguf; quantized tensors run on llama.cpp's own kernels (ggml-cuda on the GPU, ggml-cpu for hybrid / cpu experts). Supported GGUF archs:
- qwen2 (Qwen2 / Qwen2.5), qwen3 and qwen3moe (Qwen3 dense and MoE)
- qwen35 and qwen35moe (Qwen3.5 / Qwen3.6 dense and MoE, Qwen3.8-27B), qwen4exp (Qwen3.8-Flash-Next)
- gemma4 (Gemma-4 12B, 26B-A4B, 31B)
- llama (Llama 3.x, and Mistral files converted before llama.cpp's mistral3 arch)
- muse-glimmer (Muse-Glimmer)
- minimax-m2 (MiniMax-M2 / M2.x), minimax-m3 (MiniMax-M3)
- glm4moe (GLM-4.5, GLM-4.5-Air, GLM-4.6, GLM-4.7), glm5-next (GLM-5.3-Flash)
- Support new models.
- DeepSeek-V4.1-Flash, text and images (#603)
- Image input for DeepSeek-V4-Flash-Vision-Exp (#502)
- Faster NVFP4 on sm120 (RTX 50xx, RTX PRO Blackwell).** NVFP4 checkpoints with static activation scales run on NVFP4 activations (W4A4): at 128 or more tokens per expert the MoE layer is 2.54x faster on the RTX PRO 6000 and 3.72x on the RTX 5090 (#736), and Qwen3.8-27B-NVFP4 and Gemma-4-31B-IT-NVFP4 reach the first token 1.6x to 2.7x faster at 1k to 8k prompt tokens (#737).
GGUF Support
FreeToken now serves llama.cpp GGUF checkpoints (#690). Quantized tensors keep the ggml type they have in the file and run on llama.cpp's kernels, vendored from llama.cpp 8c2134a8: ggml-cuda MMQ / MMVQ on the GPU, and ggml-cpu row kernels for experts on the CPU (#704). The model config, chat template and stop tokens come from the file's metadata.
- Weight types (per tensor; a file preset such as Q4KM or UD-Q8KXL mixes several):
- Supported: F32, F16, BF16; Q40, Q41, Q50, Q51, Q80; Q2K, Q3K, Q4K, Q5K, Q6K; IQ1S, IQ2XXS, IQ2XS, IQ2S, IQ3XXS, IQ3S, IQ4NL, IQ4XS; MXFP4, NVFP4; Q10, Q20. The layers of one file may use different types, and an expert layer may pair a float projection with a quantized one (#702).
- NVFP4 GGUFs converted from ModelOpt checkpoints carry per-tensor .scale tensors; the gemma4, llama, qwen3, qwen3moe, qwen35 and qwen35moe loaders apply them on dense linears and on every MoE strategy (#732).
- Not supported yet: IQ1M (no MMQ kernel); NVFP4 token embeddings; NVFP4 .scale tensors in qwen2, qwen4exp, muse-glimmer, minimax-m2, minimax-m3 and glm4moe files (refused at load); --dtype float16 with NVFP4 .scale tensors; TQ10 and TQ20; Q4044 / Q4048 / Q4088, which llama.cpp has removed; types defined only in llama.cpp forks such as ik_llama.cpp; expert layers whose gate and up projections use different types.
- MoE strategies: fused and offload for every type (#690); hybrid and cpu for every type on x86-64 CPUs, where ggml-cpu picks the best of llama.cpp's x86 CPU builds at startup (#704). Layers of different types get their own slot pools in the expert cache (#689, #700); cpu refuses files whose expert layers use several types. ft bench bw --model <file>.gguf measures the hybrid fetch split for each expert layer kind (#734).
- scripts/gguf_check/ in the source tree checks whether GGUF files load (config, weights, kernel coverage, chat template) from header-only stubs, without downloading the weights (#756).
- Weights past the 32-bit block offsets of MMQ / MMVQ raise instead of running (#699).
- Not supported yet: image input (mmproj files), Gemma-4 E2B / E4B, the mistral3 arch, Mixtral, GLM-4.7-Flash (deepseek2 arch), glm4moe files with a fused attn_qkv, tensor parallelism.
Supported architectures
| Family | GGUF arch | PR |
|---|---|---|
| Qwen2 / Qwen2.5 | qwen2 | #707 |
| Qwen3 dense, Qwen3 MoE | qwen3, qwen3moe | #707 |
| Qwen3.5 / Qwen3.6 dense and MoE, Qwen3.8-27B | qwen35, qwen35moe | #701 |
| Qwen3.8-Flash-Next | qwen4exp | #690 |
| Gemma-4 26B-A4B (every ggml type; was Q4_0 only) | gemma4 | #690 |
| Gemma-4 12B, 31B | gemma4 | #706 |
| Llama 3.x, Mistral (files converted before llama.cpp's mistral3 arch) | llama | #722 |
| Muse-Glimmer | muse-glimmer | #724 |
| MiniMax-M2 / M2.x, MiniMax-M3 | minimax-m2, minimax-m3 | #725 |
| GLM-4.5, GLM-4.5-Air, GLM-4.6, GLM-4.7 | glm4moe | #731 |
| GLM-5.3-Flash | glm5-next | #721 |
Model Support
- Support DeepSeek-V4.1-Flash (DeepseekV41ForCausalLM), text and images (#603)
- --swa-decoder-replay bounded|exact: bounded (default) is the tech report's decoder SWA replay, exact runs the decoder on every prompt token
- The two Engram tables stream from the checkpoint shards on demand
- reasoningeffort takes low / high / max; a 1-100 budget goes through chattemplate_kwargs; DSML tool calls are parsed
- Image input for DeepSeek-V4-Flash-Vision-Exp (DeepseekV4ForCausalLM) (#502)
- Qwen3.5 / Qwen3.6 / Qwen3.8-27B: add the zero-centred norm weights' +1 in fp32 at runtime, as transformers does, instead of folding it into the bf16 weight (#755)
- Qwen3.8-Flash-Next:
- Serve the model card's 1M-token YaRN recipe through --hf-overrides (#573)
- Reduce QSA block top-k candidates past 16 splits: the per-layer top-k of one decode step at ~985k tokens of context goes from 845 to 45 µs on an RTX 3090 Ti (#571)
- Load compressed-tensors exports (per-channel fp8 dense layers, NVFP4 experts) and bf16 PLE tables (#759)
- Fence disk PLE staging against in-flight lookups (#588)
- Keep the Muse-Glimmer attention gate as its own linear next to qkv_proj (#741)
- Preserve the GLM-4 router correction bias in fp32 (#469)
- Raise WeightLoadError when the checkpoint cannot be read (#518)
- Add scripts/checkpoint_diagnose/ to the source tree: tells whether FreeToken or the checkpoint is at fault when an HF checkpoint does not load, and fixes a non-standard local checkpoint after showing the plan (#762)
Quantization
- Keep NVFP4 weight global scales in fp32. They were converted to fp16, where many of them fell in the subnormal range (#739).
- On sm_120 (RTX 50xx, RTX PRO Blackwell), NVFP4 checkpoints with static activation scales (ModelOpt NVFP4, compressed-tensors NVFP4) run on NVFP4 activations (W4A4):
- MoE prefill from 4 tokens per expert: at 128 or more tokens per expert the MoE layer is 2.54x faster on the RTX PRO 6000 and 3.72x on the RTX 5090 (geomean over 9 model shapes) (#736)
- Linears: W4A4 prefill and new W4A16 kernels for decode; time to first token at 1k to 8k prompt tokens is 1.6x to 2.7x faster on Qwen3.8-27B-NVFP4 and Gemma-4-31B-IT-NVFP4 (#737)
- --quant-backend moe.nvfp4=triton and linear.nvfp4=triton keep W4A16
Engine
- Split the offload expert cache into slot pools: --moe-cache-scope unified|per-layer, --moe-cache-split equal|profile:<path>, and --moe-cache-mem to size the cache in memory (#689)
- Serve models whose MoE layers use different expert formats, one slot pool per format (#700)
- --moe-strategy fused serves NVFP4 and MXFP4 experts through the expert banks (#601)
- Serve block-FP8 experts on the cpu and hybrid strategies (#662)
- ft bench bw --model <checkpoint> benches every MoE layer kind of a GGUF or HF checkpoint, and hybrid takes its fetch split per layer (#734)
- Page-locked expert banks: when pinned memory is capped, hybrid with --moe-cpu-layers OS-locks the expert banks past the pin limit, and the locked layers keep GPU slots and fetch part of their misses through a pinned staging buffer. This also works for models whose expert layers need several slot pools, such as GGUF files that mix expert types. Adds --moe-locked-fetch, --moe-locked-max-fetch and --moe-collect-stats (#742)
- Spin-wait the CPU MoE pool between layers: +20% batch-1 hybrid decode on Qwen3.6-35B-A3B-NVFP4 (H100) (#619)
- Split hybrid decode routes inside the ensure kernel (#685)
- Split the CPU MoE executor into weight formats and GPU sync backe
v0.1.32026-09-16
Highlights
- Image input. Qwen3-VL, Qwen3.6, Qwen3.8-Flash-Next, Gemma-4, GLM-5.3-Flash, MiniMax-M3 and Muse-Glimmer-30B accept images through the OpenAI, Anthropic and Responses APIs. Vision towers stream their weights from host memory; --text-model-only serves a multimodal checkpoint as text.
- Support new models.
- Qwen3.8-Flash-Next (#257)
- GLM-5.3-Flash (#332).
- Quantization rebuilt (#418). One QuantConfig → QuantScheme → kernel path for every layer; kernels are selected with --quant-backend layer[.kind]=name, and --moe-backend becomes --moe-strategy.
Multimodal
FreeToken now serves vision-language models (#454): an image goes through the checkpoint's own processor and vision tower, then down the same scheduler, radix cache and MoE offload path as text.
- Engine: image spans are content-keyed in the radix cache; encoded embeddings sit in an encoder cache between prefill chunks; the vision tower streams its weights from pinned host memory two blocks at a time.
- Server API: OpenAI imageurl, Anthropic image blocks and Responses inputimage (URL, base64, file://); tool-result images are carried onto the next user turn (db64879d); /v1/stats reports model.input_modalities.
- CLI: --text-model-only, --image-min-tokens / --image-max-tokens, --mm-processor-kwargs, --mm-encoder-weights, --allowed-media-domains, --allowed-local-media-path — see the CLI reference.
- ft launch: declares image input to Codex, OpenCode, OpenClaw and dsh when the server serves it (8ff0cce6).
Supported models
| Family | Arch | PR |
|---|---|---|
| Qwen3-VL | Qwen3VLForConditionalGeneration, Qwen3VLMoeForConditionalGeneration | #454 |
| Qwen3.6 / Qwen3.5 | Qwen35MoeForConditionalGeneration, Qwen35ForConditionalGeneration | #454 |
| Qwen3.8-Flash-Next | Qwen4ExpForConditionalGeneration | #454 |
| Gemma-4 | Gemma4ForConditionalGeneration, Gemma4UnifiedForConditionalGeneration | #467 |
| GLM-5.3-Flash | Glm5NextForConditionalGeneration | #479 |
| MiniMax-M3 | MiniMaxM3SparseForConditionalGeneration | #480 |
| Muse-Glimmer-30B | MuseGlimmerForConditionalGeneration | #481 |
Model Support
- Support Qwen3.8-Flash-Next (#257)
- Stream the PLE n-gram table from disk with --ple-backend disk (#311)
- Load block-fp8 dense projections natively (#428)
- Write the PLE table next to the FTW (#420)
- Support GLM-5.3-Flash (#332)
- Add scripts/ftw_hotfix.py to repair FTW checkpoints from older builds (#424)
- Serve and repair FTW checkpoints around the vision encoder (#486)
- A truncated FTW shard raises OSError instead of loading garbage (#25)
- Download the shards the safetensors index names (#336)
Quantization
Checkpoint quantization is rebuilt on a QuantConfig → QuantScheme → QuantMethod path (#418, #427): the checkpoint's quantization_config (ModelOpt, compressed-tensors, block-fp8, mxfp4) decides the scheme of every projection, and each layer declares its tensors and picks its kernel from that, instead of per-family format branches. New formats and kernels plug in once; --quant-backend layer[.kind]=name selects the kernel.
- Qwen3.5 / Qwen3.6 MoE: read every checkpoint layout through the QuantConfig (#438)
- Qwen3.8-Flash-Next: take the expert quant kind from the checkpoint QuantConfig (#426)
- Load ModelOpt exports that ship no input_scale (#462)
- Detect NVFP4 experts behind a mixed-precision compressed-tensors format (#343)
Engine
- Per-layer host-bank residency: split lock-CPU / pin-GPU layers, auto selection under a capped pin quota (#112)
- Resolve --moe-strategy auto to fused on unified-memory GPUs such as GB10 (#445)
- Reserve paged KV at allocation granularity (#367)
Kernels
- Route fused_topk through the in-repo Triton router (#319)
- Make Triton top-k/top-p sampling exact (#329)
- Compute shared experts before in-place routed experts (#463)
- Avoid the row-wise scaledmm stall on sm_89 with torch < 2.12 (#243)
- Stop l2norm recompiling per token count (f7c31e92)
- Add sm_80 (A100/A800) to the kernel-cache default arches (#75)
Server and CLI
- Add --gpu to choose the GPU on multi-GPU machines (#117)
- Honor the configured output default across APIs (#411)
Dependencies
- transformers >=5.16,<5.17 (was >=5.5,<6) (#479); torchvision and pillow added for the image processors (#454).
Breaking Changes & Deprecations
- --device is gone from ft checkpoint and ft bench bw; both take --gpu <uuid|index>, as ft serve does (#117).
- --moe-backend is now --moe-strategy; the old spelling still works as a deprecated alias. --nvfp4-backend is deprecated in favour of --quant-backend moe.nvfp4=<kernel> (#418).
- Multimodal checkpoints build their vision tower by default. Qwen3.6, Qwen3.8-Flash-Next, Gemma-4, GLM-5.3-Flash, MiniMax-M3 and Muse-Glimmer-30B now serve images out of the box; --text-model-only restores the text-only footprint (#454).
- Older FTW checkpoints may need a repair before they load. Reconvert with ft checkpoint, or patch in place with scripts/ftw_hotfix.py (see docs/ftw-hotfix.md):
- NVFP4 dense checkpoints: the FTW must carry the input_scale its scheme declares (#438)
- Qwen3.8-Flash-Next: the PLE table now lives next to the FTW (#420)
- any FTW converted before its family served images holds no vision encoder; ft serve refuses it unless started with --text-model-only (#486)
Contributors
@jason-fxz, @andy-yang-1, @taking-lying-flat, @RuixiangMa, @bernimccoy, @earlvanze, @iamanishx
First contributions this release:
- @RuixiangMa in #25
- @bernimccoy in #243
- @taking-lying-flat in #329
- @earlvanze in #411
- @iamanishx in #445
Full changelog: https://github.com/FlashML-org/FreeToken/compare/v0.1.2...v0.1.3