TensorFold 1.2.72026-11-11
TensorFold 1.2.7
1.2.7 is mostly about resuming long conversations where they left off: Qwen3.8-27B on Metal resumes a follow-up at the end of its history even when the client drops earlier reasoning (first token in about 0.3 s instead of 85 s on an M3 Ultra), a failed disk restore falls back to the state already in RAM, and GLM-5.3-Flash on CUDA can resume a chat at its previous turn (opt-in). It also adds a WebSocket mode for the Responses API and LightOnOCR-3 image chat on CUDA, with speedups on Metal, CUDA and HIP and fixes for -Dgpu builds, damaged JPEGs and malformed GLM tool calls.
- New:
- Responses API WebSocket mode: GET /v1/responses with a WebSocket upgrade, so client.responses.connect() works unchanged;
- LightOnOCR-3 0.8B and 4B serve image chat on CUDA from the publisher's BF16 weights;
- TFGLMREQUEST_STARTS=1 (opt-in) resumes a GLM chat's next turn at its previous turn's history on CUDA;
- TFFLASHNEXTSCALE_INDEX=1 (opt-in) reads Flash Next's 4-bit scales and biases through a 16-bit index on Metal (M1 through M4);
- the dashboard and /stats show the prompt cache, its disk tier and, on Metal, Flash Next's SSD-expert cache;
- a Qwen3.8-27B conformance profile for XPU on an Arc Pro B70;
- docs/optimizing.md says how to approach speed work in TensorFold.
- Speed and memory:
- Qwen3.8-27B on Metal: a follow-up from a client that drops earlier reasoning gets its first token in about 0.3 s instead of 85 s on an M3 Ultra (34K history).
- Flash Next on Metal: a copy stretch decodes 7.7% faster on an M3 Ultra; the opt-in scale index makes drafted decode 1.3% to 3.9% faster.
- Qwen3.8-27B on an M5 Ultra: sampled decode with DFlash2 is 6% faster, and five cold long prompts in a row with a 14 GiB prompt cache reach their first tokens 3.5% to 5.9% sooner in total.
- GLM-5.3-Flash with DFlash2 at TP=2 on two GB10s: four-stream decode is 3.7% faster greedy and 3.5% faster at temperature 0.7.
- Qwen3.8-27B on CUDA beside DSpine: agent turns decode about 6-7% faster at one stream on a GB10.
- HIP on gfx1151: 8K-token prompts reach their first token about 5% sooner on Flash Next and about 6% sooner on Nemotron 3.5 Lightning 30B-A3B.
- Flash Next on HIP at tensor parallel 2 on two Radeon AI PRO R9700s: each card holds 0.74 GiB less.
- Fixes:
- zig build -Dgpu=<gpu> compiles again.
- A damaged JPEG is refused instead of accepted with rows the decoder never wrote.
- A malformed GLM tool call stays as text instead of becoming a call with corrupt arguments.
- A failed round fails only its own streams, and a finished stream delivers its reply.
- Two failed disk restores in a row resume from the state in RAM, not from token 0.
- GLM-5.3-Flash thin loads with TFGLMLAYERS count only the weights they read at start-up.
Where a speedup was checked for equal output, its line says so. Some changes alter settings, refusals, memory or startup by default; they're listed under What can change for you.
Headline numbers
Each number names its machine and its PR.
- Qwen3.8-27B on Metal, follow-ups without reasoning: a client that sends earlier replies back without their reasoning now resumes a follow-up turn at the end of its last message. On an M3 Ultra with --prompt-cache-gib 5 and a 34K history, such turns resume about 34,100 tokens instead of 7,424, with the first token in about 0.3 s instead of 85 s. Clients that keep the reasoning resume as before, and replies are unchanged (#2811, fixes #2791).
- Flash Next on Metal (M1 through M4), grouped experts: a decode window of five rows or more runs its 4-bit experts grouped by expert, so rows that pick the same expert share its launch, and replies keep their tokens. A copy stretch decodes 7.7% faster on an M3 Ultra (256.6 to 276.3 tok/s); prose rounds, four rows or fewer, are unchanged (#2784, part of #2764).
- Flash Next on Metal, scale index (opt-in): with TFFLASHNEXTSCALE_INDEX=1, on an M3 Ultra with the MLX 4-bit checkpoint, drafted decode on one stream is 1.3% to 3.9% faster (prose 137.3 to 139.1 tok/s, a copy-heavy edit 262.8 to 273.2) and "draft": false 0.8% to 2.2% faster, with every token unchanged. It costs 7.3 GiB more in use and about 4 s more load (#2760, part of #2751).
- Qwen3.8-27B on Metal, sampled top-k: on an M5 Ultra with Qwen3.8-27B MLX 4-bit and its DFlash2 drafter, sampled verify rounds take 1.6 ms less and sampled decode is 6% faster, with the same candidates and replies; greedy is unchanged. The Qwen 3.5/3.6 CUDA engine shares this host draw for each first token and for grammar-masked windows; CUDA is not measured (#2507).
- Qwen3.8-27B on Metal, keep-warm while a request runs: on an M5 Ultra with a 14 GiB prompt cache filling memory, five cold 24K- to 64K-token prompts in a row reach their first tokens 3.5% to 5.9% sooner in total; with the cache off, cold prompts are level. Replies are unchanged (#2725, fixes #2724).
- GLM-5.3-Flash on CUDA with DFlash2, at TP=2 on two GB10s: with distinct prompts, four-stream decode is 3.7% faster greedy and 3.5% faster at temperature 0.7. One stream, the MTP head and every reply's tokens are unchanged (#2793, fixes #2720).
- GLM-5.3-Flash on CUDA, request starts (opt-in): on two GB10s at TP=2 (EXL3 TR3, DFlash2), a chat of about 58.7K tokens resumed its later turns at 58,735 instead of 57,344, with replies identical to --prompt-cache-gib 0 and to the switch off (#2404, part of #2396).
- Qwen3.8-27B on CUDA beside a DSpine drafter, on a GB10: the dense model copies from its context after 12 matching tokens instead of 20 (DFlash2 keeps 20), so agent turns decode about 6-7% faster at one stream, with chat and code level and greedy replies unchanged (#2703, after #2633).
- HIP on gfx1151, routed prompt tiles: Flash Next's 8K-token prompts reach their first token about 5% sooner, and Nemotron 3.5 Lightning 30B-A3B's about 6% sooner. Served Qwen3.5/3.6 MoE prompts are unchanged, and replies are unchanged (#2576).
- Flash Next on HIP, on two Radeon AI PRO R9700s at tensor parallel 2 with a 32k window: each card holds 0.74 GiB less, and --policy prefill_step=2048 loads where main refused it, with replies equal to main's (#2670).
New models and backends
- LightOnOCR-3 on CUDA: the 0.8B and 4B models serve image chat on a GB10 from the publisher's BF16 weights. A plain BF16 Qwen3.5 checkpoint with an image tower the CUDA tower reads and no Clef joint head now opens as chat, its tied head packed from the embedding. The flags it is not qualified on (--drafter, --slide, --kv-cache fp8, --prompt-precision int8), towers the CUDA tower cannot read and a joint head beside a tied backbone are refused by name before the weights load. MLX, EXL3, ModelOpt and Clef checkpoints load and answer as before (#2597, part of #1927).
- OCR kernel groundwork on CUDA (part of #1927): no model serves through these in this release:
- tfdotsqkv_rope adds dots.mocr's q, k and v biases and rotates its queries and keys in one pass at Transformers' BF16 rounding points, with a contract against FP64 up to 1,024 rows and positions up to 131,071 (#2709);
- pattn_kernel gains instances at two query heads a block for head dims 128 and 64, the prompt attention that dots.mocr, GLM-OCR and Nemotron Parse need on CUDA (#2609).
- XPU conformance: a Qwen3.8-27B profile (tools/conformance_profiles/qwen27-xpu.json: GPTQ 4-bit group 128 with its DFlash2 drafter, four streams on an Arc Pro B70). Its first turn carries a 600-item document because the XPU prompt cache keeps states only at multiples of 64 tokens (#2677, landed as #2690).
Speed and memory
- GLM-5.3-Flash on CUDA, request starts (opt-in): TFGLMREQUESTSTARTS=1 cuts prompt passes at a request's message starts, so a chat's next turn resumes at its previous turn's history instead of the last 2,048-row point. GLM refuses it at startup with `TFK
TensorFold 1.2.62026-11-10
TensorFold 1.2.6
1.2.6 makes the prompt cache hold up under many agents at once: long conversations survive 8 to 16 agents sending follow-ups together, and a follow-up's first token comes far sooner, with every reply unchanged. Qwen3.8-27B on CUDA can draft with Tencent's DSpine-b16 (opt-in), and Qwen3.8-Flash-Next and the Qwen DeltaNet prompts get faster on HIP. Spark 2.5 and Dory load faster on CUDA with direct reads, a new --unix-socket serves on a local socket, and POST /v1/images/generations serves an engine's image capability. Settings are more consistent: blank values read as unset across GLM, Gemma 4, Flash Next and the KV share switches, and GLM's 0/1 switches are refused by name when malformed.
- New:
- Qwen3.8-27B on CUDA drafts with DSpine-b16 when --drafter names it;
- --unix-socket PATH serves on an owner-only Unix socket;
- tool-call ids are reproducible for a replayed request;
- tools/conformance.py runs a family's served exactness gates from one profile, with a Qwen3.8-27B Metal profile;
- HIP builds the DFlash2 drafter's kernels for qwen3_5 (not launched yet);
- POST /v1/images/generations for an engine with an image capability;
- --instruct-sampling card serves the model card's instruct sampling with thinking off.
- Speed:
- Many agents at once: a follow-up's median first token at 16 agents drops from 546 s to 15.9 s on an M3 Ultra and from 105 s to 25.8 s on a GB10.
- Qwen3.8-Flash-Next on HIP (gfx1151): cold prompts finish 29 to 41% sooner.
- Qwen DeltaNet prompts on HIP (gfx1151): 4.6-6.0% less time on Qwen3.8-27B and 5.7-9.9% on Qwen3.6-35B-A3B.
- Qwen3.6-35B-A3B MoE prompts on a GB10: 9.9-14.1% less time, from 2K to 64K.
- Spark 2.5 and Dory on a GB10: cold loads take 2.53 s instead of 5.03 s and 3.79 s instead of 4.79 s.
- GLM-5.3-Flash at TP=2 on two GB10s: prompts fill 13-23% faster, and one-stream code decodes 32% faster.
- Qwen3.8-27B on Macs: sampled decode with DFlash2 is faster on an M5 Ultra and an M3 Ultra.
- Flash Next on Metal: served decode is 2.4-6.0% faster on an M3 Ultra.
- Serving:
- JSON tool calls cut by max_tokens come back as calls.
- Dory on CUDA keeps prompts as pool pages with TFKVSHARE=n.
- Flash Next on Macs no longer fails a request with HTTP 500 after idle.
- Drama 3 on CUDA opens again.
Where a speedup was checked for equal output, its line says so. Some changes alter settings, refusals, memory or startup by default; they're listed under What can change for you.
Headline numbers
Each number names its machine and its PR.
Release reads (1.2.5 against 1.2.6 on fresh servers, alternating; replies equal to 1.2.5 on every machine):
- Qwen3.8-27B on an M3 Ultra: prompts 1.6 to 4.7% faster (4.7% at 64K); decode and 1-8 stream sweeps level.
- Qwen3.8-27B on an M5 Ultra: prompts up to 2.6% faster (at 64K); decode and sweeps level.
- Qwen3.8-27B on a GB10: level throughout.
- Qwen3.8-Flash-Next on a GB10: level throughout.
- Nemotron 3.5 Lightning on a GB10: level. One cell, eight streams sending the same greedy prompt, read 5.7% slower; with a prompt per stream the two releases are within 0.1%, so it was identical replies running in step, not a change.
- Prompt cache with many agents, Qwen3.8-27B with DFlash2: each agent sends one ~17K-token prompt, then a follow-up on the same conversation. Every agent's replies equal main's, at every agent count (#2699).
- On an M3 Ultra (default budget): a follow-up's median first token drops from 50.1 s to 1.42 s at 8 agents and from 546 s to 15.9 s at 16. Follow-up decode rises from 15.4 to 79.5 tok/s and from 5.9 to 88.5.
- On a GB10: from 14.6 s to 7.7 s at 8 agents and from 105 s to 25.8 s at 16. Follow-up decode rises from 47.7 to 93.9 tok/s and from 33.6 to 100.8.
- At 1 and 4 agents and on Qwen3.8-Flash-Next it's level with main, and first-turn speed is unchanged.
- Qwen3.8-Flash-Next on HIP, on a Radeon 8060S (gfx1151): sparse attention runs a new kernel with the same sums. Cold prompts of 2,048 to 63,728 tokens finish 29 to 41% sooner, with replies unchanged. Other HIP chips take the kernel but weren't run (#1877).
- Qwen 3.5, 3.6 and 3.8 on HIP, on gfx1151: the DeltaNet prompt recurrence runs at half a value head a block, with the same bits. A prompt takes 4.6-6.0% less time on Qwen3.8-27B and 5.7-9.9% less on Qwen3.6-35B-A3B (2K to 32K) (#2582, piece J of #1667).
- Qwen3.5/3.6 MoE on CUDA, on a GB10: router calls of 257 rows or more run on the shared register tiles, with every logit's bits unchanged. Qwen3.6-35B-A3B NVFP4's cold prompts take 13.4% less time at 2K (0.382 to 0.331 s), 14.1% at 8K, 11.8% at 32K and 9.9% at 64K (15.899 to 14.323 s), with decode and replies unchanged. tf-cuda-test moe-router-bench measures the crossover on other GPUs (#2537, #2499).
- Tokenizer memo, on an M5 Ultra: segments of 64 bytes or more are kept, not 1,024, with the same ids. On real agent conversations (Qwen3.8-27B tokenizer, up to 60K tokens), a warm encode of the eight longest takes 146-151 µs instead of 680-691 µs, and a one-off prompt is no slower (#2509).
- Spark 2.5 and Dory on CUDA, on a GB10: loads use direct reads by default, with the same replies:
- Spark X2.6 4B 8-bit: cold loads go from 5.03 to 2.53 s; resident memory peaks at 6.1 GiB instead of 9.4, and 0.4 GiB of page cache is filled instead of 4.2;
- Dory NVFP4: from 4.79 to 3.79 s; resident stays at 7.2 GiB, and 1.3 GiB of page cache is filled instead of 3.1 (#2449).
- GLM-5.3-Flash on CUDA, at TP=2 on two GB10s: decode windows reach 32 rows. Prompts of 2K and 8K tokens fill 23% and 13% faster, and one-stream code decodes 32% faster, with every reply equal to before (#2257).
- Qwen3.8-27B on CUDA with DSpine-b16, on a GB10 (opt-in): every drafted reply equals the same request with drafts off. Against DFlash2, agent-shaped requests decode 2.4-19.8% faster, and chat and code 10-19% slower, which is why it is opt-in (#2633, refs #2592).
- Qwen3.8-27B on Macs, sampled with DFlash2:
- a verify window draws its rows on up to four host threads. One-stream sampled decode is 4 to 8.8% faster on an M5 Ultra, with tokens and errors unchanged (#2644);
- the drafter waits only for its taps and head row. One stream on an M3 Ultra: chat 75.3 to 77.0 tok/s, code 106.8 to 109.8, with every token unchanged (#2646).
- Qwen3.8-27B on Metal: a prompt pass that keeps no row's logits skips its last layer's attention, MLP and final norm, with the prompt state and tokens unchanged. Served prefill is 1.6% faster at 4K and 2.3% at 16K on an M3 Pro, and 1.2% at 8K and 1.5-1.6% at 16K on an M5 Ultra (#2436, refs #2229).
- Flash Next on Metal, on an M3 Ultra: 6- and 8-bit row products at 2 to 4 rows take two outputs a simdgroup. A 2,560 × 6,144 product at 2 rows takes 23.5 to 28.7% less time across the three widths measured, and served decode is 2.4-6.0% faster, with every token hash unchanged (#2593, fixes #2553).
- Dory on CUDA with TFKVSHARE=n, on a GB10 at a 4 GiB cache: a 25-turn agent loop to 34K tokens spends 10.7% less time to first tokens (about level up to 12K). 72 round-robin turns keep 29 states against 9, with the same replies (#2430, part of #1657).
- --learn at 100,000 learned states, on an M3 Ultra: find, has and touch take 0.02-0.04 µs instead of 24-57 µs, and a reload's finds after another writer appended take 1.1 ms instead of 2.8 s, with the same records, order, clock and eviction victims (#2351).
New models and backends
- DSpine-b16 on CUDA (opt-in): Qwen3.8-27B drafts with Tencent's DSpine-b16 when --drafter names its folder, never by default. Its code bridge and exact-sender table run on affine-4 projections. The draft tree walker moves into core/lanes (#2633).
- HIP drafter groundwork for qwen35 (#1563):**
- the engine builds the DFlash2 drafter's kernels (tap gathers, key ring, dynamic conv, bl
TensorFold 1.2.52026-11-08
TensorFold 1.2.5
1.2.5 brings LFM2.5-24B-A2B on CUDA, Nemotron Parse and LightOnOCR-3 on HIP, and Clef's joint schema head on CUDA. It also adds TensorFold Web, a local web app. Qwen3.8-27B's MLX 8-bit checkpoints get faster prompts on Macs, and a new opt-in keeps its keys and values in pool pages. HIP's opt-in gfx12 matrix tiles get faster with the same bytes. Settings are stricter: DeepSeek-V4.1's settings are refused by name when malformed, and an explicit thinking off that a chat template can't honour is refused.
- New:
- LFM2.5-24B-A2B on one NVIDIA GPU, from the MLX 4-bit export;
- NVIDIA's Nemotron 3.5 Parse on HIP;
- LightOnOCR-3 (0.8B, 1B, 4B) on HIP;
- Clef's joint schema head and /v1/systemone on CUDA;
- TensorFold Web, opened with tensorfold web;
- the HIP qwen3_5 engine reads a DFlash2 drafter's config and weights, and taps its target layers;
- a CPU projection and RMS norm in the kernel contract.
- Speed:
- Qwen3.8-27B MLX 8-bit on an M5 Ultra: cold prompts are 18% faster and decode 7.6% faster.
- Flash Next BF16 cold prompts on a GB10 take about 4% less time, and under 1% less again on top.
- HIP on gfx12 with --policy matrix=on: reranking and embedding take 2.9-4.8% less time on an R9700, with the same bytes.
- Qwen 3.5 and 3.6 on HIP: sampled streams with an MTP head make 4-12% more tokens a second.
- Qwen3.8-27B prompt chunks on an M3 Pro: 2-3% faster served prefill, bit-equal.
- --learn with 100,000 kept states starts about 5 s sooner.
- Serving:
- An explicit thinking off that the template can't honour is a 400.
- A refused disk read tries one shorter kept state.
- Qwen 3.5 and 3.6 streams on CUDA take less memory.
- A request's stats list its prompt pass widths.
- The PDF render worker no longer crashes on macOS.
- New opt-ins: the Qwen CUDA history starts with drafts; Gemma 4's 2,048-row solo chunk; CUDA prompt-cache keep-others; Qwen3.8-27B Metal KV pages.
Where a speedup was checked for equal output, its line says so. Some changes alter settings, refusals, memory or startup by default; they're listed under What can change for you.
Headline numbers
Each number names its machine and its PR.
Release reads, each machine with 1.2.4 and 1.2.5 in alternating fresh servers (base, new, new, base, after an unscored warm-up):
- Flash Next MLX 4-bit on a GB10, against 1.2.4: level, with the same tokens: every prompt and decode cell from 2K to 64K within 0.8%, and every sweep from one to eight streams within 1.1% apart from greedy chat at four streams (3.6% faster). The 64K long-context answer is right on both.
- Qwen3.8-27B on an M5 Ultra, against 1.2.4: level, with the same tokens: every sweep within 0.4% and every prompt and decode cell from 2K to 64K within 1.2%, apart from one 1.2.5 run's first 2K prompt, which came up cold while the other 1.2.5 run matched both 1.2.4 runs.
- Qwen3.8-27B on a GB10, against 1.2.4: level, with the same tokens: all 56 cells within -1% to +2%. The 64K long-context answer is right on both.
- Nemotron 3.5 Lightning 30B-A3B on a GB10, against 1.2.4: level, with the same tokens: prompts within 1% and one stream level. Greedy chat at two to eight streams moved both ways (-2.8% to +1.5%), as one-prompt multi-stream cells do; decode after a 32K prompt resumed 10K tokens in was 2.5% lower in both 1.2.5 runs.
- Qwen3.8-27B on an M3 Ultra, against 1.2.4: level, with the same tokens: every prompt and decode cell from 2K to 64K within 1.3%, and 2K prompts 2.1% faster in both 1.2.5 runs. Sweeps were within 0.4%, against one 1.2.4 run. The 64K long-context answer is right on both.
- Qwen3.8-27B MLX 8-bit (groups of 64), on an M5 Ultra: cold prompts are 18% faster (649 to 768 tok/s at 2K) and decode 7.6% faster, with the same tokens and prompt state (#2518, fixes #2502).
- Qwen3.8 Flash Next on CUDA, on a GB10 (NVFP4):
- BF16 prompt projections take a 128 × 128 tile on sm_12x wherever it fills the GPU, with the same output bytes: cold prompts take 3.7 to 4.4% less time (4K to 34K tokens), replies unchanged (#1846, refs #1754).
- Staging K32 fragments from the same K64 pack takes another 0.6 to 0.8% off cold prompts, with the same bytes (#1848).
- HIP on gfx12 under --policy matrix=on, on a Radeon AI PRO R9700, with the same output bytes:
- the WMMA dense tile's 64-deep K step: Qwen3-Reranker-8B ranks 16 documents in 0.694 s instead of 0.720 s, and Qwen3-Embedding-8B embeds 100 short inputs in 0.946 s instead of 0.974 s (#2280, part of #2245);
- silu(gate) × up in the product's epilogue, on top: 0.657 to 0.661 s instead of 0.687 to 0.697 s, and 0.901 to 0.911 s instead of 0.940 to 0.948 s (#2344, part of #2245).
- Qwen 3.5 and 3.6 on HIP, on gfx1151: sampled streams with an MTP head draft on the graphed head chain, with the same tokens: 4-12% more tokens a second at temperature 1.0 (#2455).
- Qwen3.8-27B on Metal, on an M3 Pro: prompt chunks of 64 rows or more use half operands where a tile's inputs fit. Logits and tokens are bit-equal, and served prefill is 2-3% faster (#2435, part of #2089).
- Qwen CUDA with TFQWENREQUESTSTARTS=1, on a GB10 with Qwen3.8-27B 4-bit and its DFlash2 drafter:** turns 2 to 6 of a six-turn chat reuse 1,450 to 1,635 prompt tokens and take 0.72 to 1.33 s instead of 1.78 to 2.10 s (#2176, carries #1335).
- Gemma 4 on CUDA with TFGEMMA4CHUNKSOLO=2048, on a GB10 with the 31B 4-bit release:** a cold prompt's first token comes 4-5% sooner at 4K, 16K and 29K tokens, with the same replies (#2391).
- Gemma 4 and Nemotron on CUDA, on a GB10: prompt chunks that nothing reads skip their last layers, so cold prompts start up to 3.9% sooner (Gemma 4 31B: 1.3% at 2K, 3.1% at 16K, 3.9% at 32K; Nemotron 3.5 Lightning level at 2K, about 2% sooner at 16K and 32K), with identical replies (#2432, refs #2165).
- Qwen 3.5 and 3.6 on CUDA, on a GB10: each pooled stream takes 147 MiB less on Qwen3.8-27B and 61 MiB less on the 35B-A3B, with tokens and speed unchanged (#2447, part of #1943).
- --learn with 100,000 kept states, on an M3 Ultra: the server starts about 5 s sooner (8.0x faster open), with the same states and files (#2316).
New models and backends
- LFM2.5-24B-A2B on CUDA: served on one NVIDIA GPU from the MLX 4-bit export, on the shared CUDA kernels, with --parallel streams and copy drafts. It runs without a drafter or prompt cache (#1492, part of #1488).
- Nemotron Parse on HIP: NVIDIA's nvidia/Nemotron-3.5-Parse is served natively from its FP32 safetensors on a Radeon AI PRO R9700 (gfx1201), with its RADIO image tower, convolution neck and mBART decoder. The token arrays match the published example's on three pages, plain and drafted, solo and concurrent (#1941, part of #1927).
- LightOnOCR-3 on HIP: the native server reads lightonai/LightOnOCR-3-0.8B, -1B and -4B (BF16) and serves their OCR prompts, token-exact against Transformers apart from exact BF16 logit ties (#1965, for #1927).
- Clef's joint schema head on CUDA: /v1/systemone answers on one NVIDIA GPU, as it already did on HIP (#2458).
- Its readout streams fill in pieces of final-norm rows, 1,024 rows a piece unless the lane host caps it, as on HIP (#2384, part of #2324).
- TensorFold Web: a local web app laid out like the desktop app (chat, models, learning, API, benchmarks and clusters) ships in web/.
- tensorfold web opens it, and on first use offers to install it with uv, pinned to the same release.
- It starts without network access and reaches a local engine directly, past any system proxy.
- Adopted from @Stephane-Eden's TensorFold Studio (#1781).
- HIP drafter groundwork for qwen35 (#1563):**
- the engine captures a drafter's requested target-layer taps in each stream's ring, with and without graphs (#2314);
- it reads a DFlash2 drafter's config.json against its target, refusing one it can't run, and load
TensorFold 1.2.42026-11-07
TensorFold 1.2.4
1.2.4 brings Gemma 4 E4B on CUDA, Cloudflare's Clef-Flash on HIP, LFM3-1.7B's text decoder on Metal, and the HIP runtime on Windows. DeepSeek-V4.1 recalls a saved long conversation in seconds instead of minutes. Flash Next gets faster prompts and decode, sooner first tokens, kept prompt states with drafts at --tp 2, faster SSD experts, and much faster overlapping clients on M5 Macs. Gemma 4 and Qwen 3.5 load faster cold on CUDA. HIP decode and rerank are faster. GLM on two or more GB10s prefetches its next decode weights into L2. DeepSeek-V4 and V4.1 prompts follow the release's recorded encoder outputs. An engine's refusals now reach the client as clear 400s.
- New:
- Gemma 4 E4B (NVIDIA Nemotron 3.5 Bolt E4B) on CUDA;
- Clef-Flash's joint schema decisions on /v1/systemone on HIP;
- DeepSeek-V4.x chat rendered with its own encoder;
- LFM3-1.7B text on Metal, from the official MLX checkpoints;
- the HIP runtime on Windows;
- Clef-Flash's BF16 Qwen3.5 backbone on CUDA, as a decision backbone.
- Speed:
- A 319K-token DeepSeek-V4.1 conversation restores from NVMe in 1.2 to 1.4 s instead of 155 to 178 s.
- A Flash Next chat resumed at --tp 2 with drafts replies in 1.5 s instead of 21.2 s.
- Flash Next prompts at --parallel 1 on one CUDA GPU are 15-23% faster, and 8 streams decode 3-6% faster.
- A new short Flash Next prompt's first token on CUDA comes 19% sooner.
- Two overlapping Flash Next clients on an M5 Ultra finish in 47 s instead of 127 s.
- Gemma 4 and Qwen 3.5 on CUDA load 28-31% faster cold.
- A lone "draft": false Flash Next reply on Metal is 17-18% faster.
- Rerank on HIP is about 25% faster, and one-stream HIP decode up to 2.7% faster.
- Qwen 3.5 first tokens on a GB10 come 1 to 3% sooner.
- GLM on CUDA prefetches a decode window's next weights into L2 on two or more GB10s, with the same replies.
- Serving:
- An engine's refusal reaches the client as a 400 before any header.
- Flash Next on Metal enforces structured output at --parallel 1 too.
- Unsupported penalties are refused instead of dropped.
- A refused prompt-cache restore tries a shorter kept state.
- DeepSeek-V4.1 tool calls stream sooner.
- Tool calls made just before the thinking close are returned.
- DeepSeek-V4 and V4.1 prompts match the release's recorded encoder outputs, and malformed tool requests are refused with a 400.
- An opt-in queue-wait histogram for native metrics.
- /v1/models lists what each model supports.
- An opt-in drift gate for Living Weights lessons.
- Malformed TF, TENSORFOLD and TENSORFOLDPREFILL settings stop the server at startup, by name.
- A kept prompt state on disk is reused only by the same build and checkpoint files.
Where a speedup was checked for equal output, its line says so. Some changes alter scheduling, memory, loading, prompts or startup by default. They're listed under What can change for you.
Headline numbers
Each number names its machine. Where replies were checked against the base, the line says how.
Release reads, each machine with 1.2.3 and 1.2.4 in alternating fresh servers (base, new, new, base, after an unscored warm-up):
- Flash Next MLX 4-bit on a GB10, against 1.2.3: prompts and one stream level, with the same tokens; two to eight streams up to 5.1% faster (eight streams 1.7% to 5.1%). The 64K long-context answer is right on both.
- Qwen3.8-27B on an M5 Ultra, against 1.2.3: level, within 1.2% in every sweep and on every prompt and decode cell from 2K to 64K, with the same tokens.
- Qwen3.8-27B on a GB10, against 1.2.3: level, within 1.0% in every sweep and on every prompt and decode cell from 2K to 64K, with the same tokens.
- Qwen3.8-27B on an M3 Ultra, against 1.2.3: level on every prompt and decode cell from 2K to 64K, with the same tokens. At 64K one of the two 1.2.4 runs was up to 2.5% lower and the other matched both 1.2.3 runs. Sweeps were within 1.3%, against one 1.2.3 run.
- Nemotron 3.5 Lightning 30B-A3B on a GB10, against 1.2.3: level, with the same tokens; no cell slower than 1.5%, and decode after a 32K prompt 2.0% to 2.6% faster.
- DeepSeek-V4.1 on CUDA, on two GB10s at TP=2, with TFDSV41PREFILL=replay:
- recalling a 319K-token conversation from the NVMe session tier takes 1.2 to 1.4 s instead of 155 to 178 s;
- replies are byte-identical to a cold prefill (#2182).
- Flash Next on CUDA at --tp 2 with drafts, on two GB10s: a chat with about 28,500 tokens of history replies in 1.5 s resumed, against 21.2 s fresh, with the same replies (#2214).
- Flash Next on CUDA, on a GB10:
- at --parallel 1, prompts run 15-23% faster, for about 3.2 GiB more memory (#2246);
- 8 streams decode 3-6% faster with the same tokens (#1942);
- a new short prompt's first token comes 19% sooner, with replies unchanged (#1866).
- Gemma 4 and Qwen 3.5 on CUDA, on a GB10: loading cold is 28-31% faster, with the same weights and replies, for about 0.6 GB more host memory (#2282).
- Flash Next on Metal, on an M5 Ultra with the MLX 4-bit checkpoint, at --parallel 2 or more: two overlapping clients finish in 47 s instead of 127 s, and a 64K prompt beside a 1,024-token reply in 21 s instead of 36-38 s, with the same tokens (#2394).
- Flash Next on Metal, on an M3 Ultra: a lone "draft": false reply at --parallel above one is 17-18% faster, with the same tokens (#2272).
- Qwen3-Reranker-8B on HIP, on a Radeon AI PRO R9700: 16 documents of 323 tokens, 87 of them shared, take:
- 1.98 s instead of 2.63 s on the default dot2 tile;
- 0.73 s instead of 0.99 s with the gfx12 WMMA tile (--policy matrix=on).
- The responses are the same byte for byte on that card (#2262). On RDNA 3.5 (gfx1150 to gfx1153), a prompt row now gets the same bytes wherever its chunk starts (#2357).
- HIP decode, on a gfx1151: one stream decodes 2.7% faster on Qwen3.6-35B-A3B and 1.1% on the 27B, with the same output bytes (#2247).
- Qwen 3.5 on CUDA, on a GB10:
- first tokens come 1 to 3% sooner, with the last row's logits bit-equal (#2229);
- 8k and 32k prompts start 1.6 to 2.1% sooner with the same replies (#2113).
New models and backends
- Gemma 4 E4B on CUDA (NVIDIA Nemotron 3.5 Bolt E4B): served from mlx_lm.convert 4-bit checkpoints.
- Per-layer inputs run in their own launch, and layers 24 to 41 read the caches of layers 22 and 23.
- Replies are equal across streams and resumes (#2230, part of #1222).
- Cloudflare's Clef-Flash on HIP (Radeon AI PRO R9700): its BF16 Qwen3.5 backbone and joint schema head answer /v1/systemone.
- The head runs on the GPU over each prompt piece in the shared rounds.
- Validation errors are 422s with their location.
- Chat and the other routes refuse the checkpoint, naming /v1/systemone.
- capabilities --json lists BF16 for HIP Qwen3.5 under a new decisions key; the chat formats are unchanged.
- Checkpoints whose tokenizerconfig.json publishes a pretokenizeregex split words with it (#2048, #1360, #1952).
- DeepSeek-V4.x chat on the native server renders with its own encoder and schema prompt.
- A served image part becomes the image token.
- The startup line names the chat renderer.
- Prompt token ids now equal the release's encoder goldens (#1698).
- LFM3-1.7B on Metal: the text decoder serves the official MLX 4-, 6- and 8-bit checkpoints, with per-stream convolution and KV state, verification windows, snapshots and the prompt cache. LFM3-VL checkpoints are refused by name until the vision tower lands (#1514, part of #1491).
- HIP on Windows: the HIP runtime builds and loads on Windows.
- hipcc.exe is preferred, and hipcc.bat is found through -Dhipcc, PATH or HIPPATH/ROCMPATH and run through cmd /c call.
- The runtime and the optional libraries load by their Windows names (amdhip64.dll and the versioned ones; TENSORFOLD_HIP
TensorFold 1.2.32026-11-05
TensorFold 1.2.3
1.2.3 is mostly fixes, memory and serving work, with a few new paths.
- HIP: dots.mocr, Qwen3-Embedding-8B and Qwen3-Reranker-8B are served natively, and Qwen 3.5 loads BF16 checkpoints through rocBLAS.
- Metal: the Qwen3.8-27B engine serves Qwen3.5-MoE checkpoints, including Ornith-1.5-122B on one M5 Ultra.
- PDF documents work as input with --vision --vision-pdf.
- Intel GPUs: LFM3-1.7B runs on Iris Xe.
- Kimi K3 and K3.1 are reachable through the native server on Metal, but only as groundwork. They have run on a synthetic graph, not on real weights; see below.
- Speed: Flash Next stops a long prompt from stalling a running reply on Macs without tensor units, and decodes faster on a GB10, at --tp 2 on two GB10s and on a Strix Halo. A GLM cold prompt at TP=3 gets its first token sooner.
- Memory and correctness: a prompt cache you size with --prompt-cache-gib now gets that size on CUDA KV pages. Cached-token counts can't exceed the prompt. A few loaders and decoders refuse bad input they used to accept.
Where a speedup was checked for equal output, its line says so. Opt-in switches change nothing unless set. A few changes move replies, change what a request returns, or refuse a start that used to go ahead. They're listed under What can change for you.
Headline numbers
Each number names its machine. Where replies were checked against the base, the line says how.
- Flash Next NVFP4 on two GB10s at --tp 2: plain decode is 21% to 23% faster, drafted decode 9% to 11% faster at one stream and 76% to 77% at two, and prompts about 6% faster. Each rank holds 2.62 GiB less of the model and a stream's KV at 65,536 tokens takes 40% less, with replies token-equal to one GPU (#1784).
- Flash Next on an M3 Ultra, --parallel above one: a 1,024-token reply beside a 64K prompt finishes in 8 s instead of 48 s. The prompt's first token comes at 54 s instead of 90 s, with the same tokens (#2167).
- Flash Next on an M3 Ultra, a lone reply at --parallel above one: it runs as fast as at --parallel 1. Sampled replies are 12% to 19% faster than before and greedy ones about 4%, with the same tokens (#2199).
- Flash Next on an M3 Ultra with --ssd-experts: about 11% less read from disk at 17 and 35 GiB pools, and a mixed workload took 8% less time at 35 GiB, with the same replies (#2209).
- Flash Next FP4 on a GB10: one-stream decode up to 11.5% faster (chat +8.7%, code +6.3% at temperature 0), with every reply token-equal to the previous version (#1786).
- Flash Next on a Strix Halo (gfx1151): the first request after start decodes at 35.8 tok/s instead of 24.4, and a cold 7K-token prompt starts in 28.9 s instead of 48.4 s, with every reply token-equal (#1615).
- HIP on a gfx1151: Qwen3.8-27B at 16 streams goes from about 51 to 75 tok/s on code, with the same bytes. These are the contributor's own records, as we have no AMD GPU (#1694).
- GLM-5.3-Flash on three GB10s at TP=3, with TFGLMROCEMULTI=1 and TFGLMCHUNKSOLO=4096: a cold prompt's first token comes about 5% sooner, with the same tokens (#2116).
- GLM-5.3-Flash on two GB10s at TP=2 with --drafter: TFGLMCHUNK_SOLO=4096 now serves, and a cold prompt's first token comes 6.1% sooner at 32K and 7.0% sooner at 64K, with the same tokens, for 80 MiB more a rank (#2219).
- GLM-5.3-Flash on an M3 Ultra: about 14% more throughput at four streams when drafts land often, with the same tokens. This was measured on 40 of the model's 45 layers, not the whole model (#2111).
- GLM-5.3-Flash csf0v1 checkpoints on CUDA:** about 13 GiB less across a TP=2 pair, with the same tokens (#2041).
- Flash Next MLX 4-bit on a GB10, against 1.2.2: one stream is 3.5% to 6.6% faster and eight streams 1.1% to 3.8%, with prompts level and the same tokens. Decode after a long prompt is faster in 18 of 20 of our cells, by 2.3% to 14.8%; two cells where drafts land very often are slower, 32K by 5.2% and 64K by 3.1%, the cost of the same draft chain rule.
- Qwen3.8-27B on an M5 Ultra, against 1.2.2: level, within 0.4% in every sweep and within 0.9% on decode after a 32K prompt, with the same tokens.
- Nemotron on a GB10, against 1.2.2: level over 18 repeats a build at four streams, the one width where the same request can swing widely between runs, on both versions.
- Qwen3.8-27B on a GB10, against 1.2.2: level, within 0.7% in every sweep and on every prompt and decode cell from 2K to 64K, with the same tokens.
New models and backends
- dots.mocr on HIP: the publisher's BF16 dots-studio/dots.mocr OCR checkpoint is served natively, with its own template, tokenizer, image processor and EOS ids. It's token-exact against its model card's example (#1938, part of #1927).
- LFM3-1.7B on Intel GPUs through XPU, Iris Xe first (#1696).
- Qwen3.5-MoE on Metal: the Qwen3.8-27B engine serves qwen35moe checkpoints: Ornith-1.5-122B on one M5 Ultra from the publisher's own 4-bit MLX weights, with routed and shared experts, its DFlash2 drafter, images, the prompt cache and several streams. The dense 27B is unchanged (#1125).
- Qwen3-Embedding-8B on HIP: served as Qwen publishes it (BF16) through /v1/embeddings on a Radeon AI PRO R9700 (gfx1201). Inputs of up to 32,768 tokens batch to the same vectors as alone, and dimensions takes any width from 32 to 4,096 (#1978).
- Qwen3-Reranker-8B on HIP: POST /v1/rerank answers with results (index, relevancescore, document), and takes topn and return_documents. Each document's prompt is the model card's template, token for token (#1981).
- GGUF, groundwork: Qwen3.8-27B on Metal gains a host adapter for GGUF metadata, tensor names and exact affine preparation. GGUF serving remains unavailable (#2064, #1920).
- Qwen 3.5 on HIP loads BF16 checkpoints with their Hugging Face tensor names, and runs the BF16 projections through rocBLAS. MLX checkpoints are unchanged (#2046).
- Nanbeige4.3-33B-INT4 on Metal: Nanbeige's own int4 release (compressed-tensors, groups of 32, 21.0 GiB) is served as published, read in place. Any other quantization of the family is refused by name. Against a float64 reference over 1,500 teacher-forced positions on an M5 Ultra, the engine's mean KL is 0.0044 with 97.1% top-1 agreement. Prompts need Metal 4 tensor units, so M1 to M4 Macs are refused at load (#1782, part of #1414).
- Kimi K3 and K3.1 on Metal, groundwork only (#2221, #2192):
- The native server renders chat, reasoning and tool calls for both.
- They have run only on a synthetic four-layer graph. No real weights have run yet, and token ids against the release's tokenizer are still to be checked.
- The full 1.45 TB checkpoint doesn't fit one Mac through this path.
- K3.1 is capped at 2,047 tokens until its sparse index is qualified. A longer request is refused with that limit's message.
- A K3.1 config's dsa_config is checked, and refused where unsupported, instead of being read as K3.
- Images, external draft heads, TP, learning, disk recall, structured-output enforcement and budgets are refused by name.
- The native tokenizer reads a Qwen2 checkpoint's slow-tokenizer files when it ships no tokenizer.json. Structured output still needs tokenizer.json (#2225).
- Clusters: the fabric layer can link nodes over ConnectX (RoCE v2), the first step of #1594 (#1957).
- Clusters, groundwork: the fabric layer gains a pipeline stage link, a TCP ring between ranks on Linux and macOS that carries each round's plan and hidden rows. Each link proves a shared key both ways before any frame. Nothing serves through it yet (#2012, step 1 of #2005).
Speed
- Flash Next:
- At --tp 2 on CUDA, each rank computes its own output columns of the dense projections instead of both repeating the whole projection, with the same replies as one GPU. The numbers are under Headline numbers (#1
TensorFold 1.2.22026-11-03
TensorFold 1.2.2
1.2.2 adds new models on CUDA and HIP. Qwen3.8-Flash-Next serves NVIDIA's own NVFP4 checkpoint at --tp 2. GLM-OCR runs
natively on HIP, and Qwen3.5-9B runs on Metal on the 27B's engine. Qwen3.8-27B EXL3 checkpoints prefill 38% to 42% faster on a
GB10, HIP decode is faster on Ryzen AI Max, and Flash Next reads its per-layer embeddings from an SSD much faster on Macs.
CUDA and Metal loaders release the page cache once the weights are loaded. CUDA loaders read checkpoints ahead, so they load
sooner. zig build -Dgpu=<gpu> builds a much smaller server for one GPU. Readers of safetensors and other checkpoint headers
across the engine now refuse a malformed file instead of crashing on it.
Where a speedup was checked for equal output, its line says so. Opt-in switches change nothing unless set. A few changes
move replies or refuse a start that used to go ahead; they're listed under
What can change for you.
Headline numbers
Where a number's replies were checked against its base, its line says how.
- Qwen3.8-27B EXL3 on a GB10: prompt prefill 38% to 42% faster from 2K to 64K tokens (494 to 705 tok/s at 2K), and decode
5% to 12% faster, with the same tokens (#1467).
- HIP decode on a gfx1151: 7.5% to 18.9% faster on Qwen3.6-35B-A3B and Qwen3.8-27B at 32K and 64K, with identical
tokens (#1932).
- Ryzen AI Max (RDNA 3.5): Qwen3.6-35B-A3B chat decodes about 18% faster with drafts on, and Qwen3.8-27B about 5%,
with the same tokens (#1855).
- Flash Next on an M5 Max with --ple-on-ssd: decode 36% to 39% faster (75.8 to 105.1 tok/s on code), and a cold 2K
prompt in a third of the time, with the same replies (#1462).
- Flash Next on a gfx1151: the first cold prompt runs in 28.55 s instead of 48.44 s, and the page cache falls from
31.5 to 2.8 GiB (#1611).
- GLM on three GB10s at TP=3: cold prompts of 32K to 167K tokens finish 5% to 14% sooner, with identical replies (#1577).
- Flash Next on a GB10: a cold start drops from about 74 s to 48 s with the direct expert load (#1901). Cold prompts
run about 4% to 5% faster, with identical tokens (#1764).
New models and backends
- nvidia/Qwen3.8-Flash-Next-NVFP4 on CUDA at --tp 2, on two GPUs, one rank a host (#1771, for #1754).
- The BF16 attention, DeltaNet and shared-expert projections, the n-gram table (E4M3 rows with one BF16 scale) and the
MTP head's E4M3 experts (128x128 block scales) load as stored.
- Each rank owns half the routed experts whole, so replies are one GPU's bytes.
- On two GB10s, drafted replies equal "draft": false (6 of 6), and concurrent replies equal their solo runs in 8 of
8 cells.
- The CUDA server serves --tp, with --rank, --master and --master-port. Families that open without ranks
refuse it by name, and --idle-unload is refused with it.
- The ranks compute and compare a cached digest of every checkpoint byte, so TENSORFOLDCHECKPOINTSHA256 is no
longer required (it still overrides the digest) (#1771).
- GLM-OCR on HIP: the publisher's BF16 zai-org/GLM-OCR, served natively. Its CogViT image encoder and GLM decoder
run in Zig and HIP, with no Python or libtorch. Every reply was checked against the model card's Transformers examples
(#1940, part of #1927).
- Groundwork for the OCR models (#1929, #1930, #1931, #1933):
- a family can declare that its replies carry no reasoning block;
- the tokenizer supports NFKC and NFKD normalizers;
- HIP has a shared rocBLAS BF16 projection, loaded as an optional library;
- the prompt cache can count repeated keyed prompts as repeats.
- No model already served decodes differently.
- Qwen3.5-9B on Metal runs on the 27B's engine, instead of being refused as an unqualified shape (#1826, part of
#503). The dense Qwen Metal loader checks config-derived projection geometry through shared structural checks, and keeps
the 9B and 27B qualification policy (part of #503).
- DeepSeek-V4.1 on Metal, part one:
- its kernels, the decoder over every layer, Engram, prompt chunks and DSpark drafts, each checked against float64
references (#1710-#1719);
- its CSA2 attention, matvecs and glue kernels, under the kernel contract (#1709, #1710, #1711);
- serving follows with #1720-#1722.
- Intel GPUs without DPAS (Gen12LP): a server built with -Dxpu-dpas=false serves Nemotron-H. It refuses the
Qwen3.8-27B runner and Gemma 4 E4B by name, and Nemotron's prompt projection takes 8 x 4 tiles (#1692).
- Groundwork:
- core/gguf.zig reads GGUF files with every length and offset checked, and repacks Q40, Q41, Q50, Q51, Q8_0,
Q4K, Q5K, Q6K, Q3K and Q2_K blocks exactly into per-group affine weights. It is the first step toward native GGUF
on Apple Silicon (#1926, part of #1920).
- Ling's grouped expert routing runs on Metal, checked against a float64 reference (#1836, part of #1247).
- MOSS-Image computes its sampling schedule natively from scheduler_config.json, bit for bit what its scheduler sets
(#1670, part of #1529).
- Living Weights on Gemma 4 and on Qwen4 dense, step 1 each: the attention backward passes on the host, checked against
independent references. Nothing served changes (#2032, #2021).
- TensorFold Starlight, the enterprise platform, lives in its own repository,
ashhart/tf-starlight, with its history (#1922, #1972).
- It's one governed OpenAI- and Anthropic-compatible endpoint for a fleet of TensorFold engines.
- It pins the TensorFold release it is tested with, and checks the engine contract against that release in CI.
Speed
- Qwen3.8-27B EXL3 on CUDA: prefill 38% to 42% faster from 2K to 64K tokens on a GB10, decode 5% to 12% faster, with the
same tokens (#1467).
- HIP decode:
- The planned decode score takes four keys a warp on long spans and two on short ones, with the same lane order and folds:
7.5% to 18.9% faster on a gfx1151 at 32K and 64K, with identical tokens (#1932).
- On Ryzen AI Max, one- and two-row decode launches use narrower stream tiles: about 18% faster on Qwen3.6-35B-A3B chat
with drafts on, and about 5% on Qwen3.8-27B, with the same tokens (#1855).
- Softmax stats and partial sums load sixteen values ahead and fold in the same order, with the same bits (#1961).
- HIP Qwen3.5-family servers price shared draft rounds at the row bucket they actually run. Qwen3.6-35B-A3B with MTP at four
streams no longer drops into a slower drafting mode, and models load faster. Tokens are unchanged (#1830, fixes #1695).
- Flash Next on HIP:
- n-gram drafts are gathered in parallel, and the checkpoint's weight pages are released after upload. On a gfx1151 the
first cold prompt runs in 28.55 s instead of 48.44 s (#1611).
- On RDNA 3.5 iGPUs (gfx1150 to gfx1153) prompts fill in 4,096-row chunks by default on one GPU, falling back to 1,024 when
the larger chunk would admit fewer streams. Tokens are unchanged (#1651, fixes #1649).
- Routing reads the checkpoint's stored BF16 router rows, with the same tokens, for about 2.5 MiB more a layer (#1614).
- Flash Next on CUDA:
- Prompt passes do less work with the same bits, so cold prompts run about 4% to 5% faster on a GB10, with identical
tokens (#1764).
- The MTP head's experts run on a call's last row only, the one row that reaches the draft, and replies are unchanged
(#1925).
- A partly kept window's DeltaNet rows are applied at the start of the next window, lane buffers are reused across
streams, and the shared expert runs on a side stream, all with the same tokens (#1763).
TFFLASHNEXTKEEPLATE=0, TFFLASHNEXTLANEPARTS=0 and TFFLASHNEXTSHARED_SIDE=0 turn each off.
- Flash Next on Macs with --ple-on-ssd: per-layer embedding rows are read from disk in batches. Decode is 36% to 39%
faster on an M5 Max (75.8 to 105.1 tok/s on code), and co
TensorFold for Mac 1.0.02026-11-02
The first stable TensorFold desktop release for Apple silicon Macs running macOS 13.3 or later. The app is signed with Developer ID and notarized by Apple.
Download TensorFold for Mac 1.0.0
- Ask TensorFold opens as a compact input bar, with explicit window selection, capture preview and available local text extraction.
- The main window closes into the background, with the TensorFold mesh logo in the menu bar.
- Connection health and the unified inbox show account state and provider errors. Google Workspace and Microsoft connections require your configured credentials and consent.
- Portuguese is included in the interface languages.
The app bundles engine 1.2.0. Engine 1.2.1 is available separately through Models → Details → Engine updates. Checking, downloading and activation are explicit actions. The app waits for local work to finish before switching engines, and you can return to the bundled engine without replacing the app or model files.
Windows and Linux builds remain previews. Cortheon remains disabled. OS automation and messaging depend on the operating system and configured services.
Validation includes 931 passing app test files, 259 native-helper checks, isolated packaged startup on both build and verification Macs, and a real-model engine 1.2.0 to 1.2.1 upgrade and rollback. The Studio independently matched all 121 compiled application files against the pinned source build.
The DMG, automatic-update ZIP and engine archive are accompanied by signed update metadata and SHA-256 checksums. This desktop tag is separate from the engine release tags. Source: aa8611f46b374cf709bed812b1355de2b1199e84.
The Mac desktop, shared Linux desktop tests and mobile checks passed in the release CI run. The Linux AppImage preview smoke test stopped because the runner lacks the required sandbox approval. No Linux installer is published with this Mac release.
TensorFold 1.2.12026-11-01
1.2.1 adds new models on CUDA. MiniMax-M3.1-Flash runs over two GB10s, Gemma 4 31B drafts with Google's MTP
assistant, and Qwen3.6-35B-A3B gets a 1,048,576-token window and a DFlash2 drafter. MOSS-Image now runs end to end on
Metal and its denoiser runs on CUDA. Decode is faster on GB10 for MLX 4-bit models, and Gemma 4 E4B reads prompts 2 to
2.6 times faster on Intel's Arc Pro B70. GLM-5.3-Flash gets opt-in ways to keep long conversations, and the server
reports its load at GET /v1/load.
Where a speedup was checked for equal output, its line says so, and the opt-in switches change nothing unless set. A
few fixes change replies on purpose; they're listed under What can change for you.
Headline numbers
Where a number's replies were checked against its base, its line says how.
- Gemma 4 31B on a GB10, with its MTP assistant: 28.3 tok/s against 12.2 undrafted at one stream (#1385).
- Qwen3.6-35B-A3B (NVIDIA's NVFP4) on a GB10, with a DFlash2 drafter: 130 tok/s at one stream on a 4.2K-token
prompt, against 99 with MTP (#1674).
- Qwen3.6-35B-A3B on one GB10 at 1M tokens: a 1,048,512-token prompt prefills in 26 minutes and finds a
passphrase placed 5% in (#1675).
- Gemma 4 E4B on an Arc Pro B70: prompts read 2.0 to 2.6 times faster, with the same logits (#1746).
- Nemotron 3.5 Lightning on a GB10: +13.3% at 8 streams, from grouped 4-bit projections (#1783).
- GLM-5.3-Flash on GB10s at TP=3 with the 1M window: a 64k message sent again resumes in about 1.7 s instead of
prefilling for 36 s (opt-in, #1732).
New models and backends
- MiniMax-M3.1-Flash on CUDA, at tensor parallel over two GB10s (sm_121, two ranks only; anything else is refused).
- It reads Mia-AiLab's published EXL3 pack as stored. That's a lossy third-party quantization: against the release,
teacher-forced over 2,406 tokens, its mean KL is 0.0788, with 89.9% top-1 agreement.
- It serves the release's DSpark drafts, kept prompt states, tools, structured output and images. One stream with
drafts ran 24 to 39 tok/s (#1388, after #1235).
- Gemma 4 31B on CUDA takes google/gemma-4-31B-it-assistant as --drafter. Drafted replies equal
"draft": false, and four concurrent replies equal their runs alone. Other assistant types are still refused by type
(#1385, for #1383).
- Qwen3.5 and 3.6 on CUDA past 262,144 positions: --rope-yarn 4, or a config with a YaRN block, raises the
window to 1,048,576. Drafted, concurrent and resumed replies equal their plain runs. Without YaRN, replies are
byte-identical to before. HIP, Metal, the DFlash2 drafters and other families refuse a scaled rope by name (#1675).
- Qwen3.6-35B-A3B on CUDA with a DFlash2 drafter: --drafter takes a DFlash2 model, such as
incoai/Qwen3.6-35B-A3B-DFlash2. The checkpoint's own directory still drafts with its MTP layer. Every drafted reply
equals its plain run (#1674).
- MOSS-Image on Metal, end to end: the prompt encoder (bf16 as published, under the kernel contract) and the image
decoder (FLUX.2 small VAE, float32) now run inside the moss_image family, and tf-moss-image writes
picture.png itself, with no torch step (#1687, #1504).
- MOSS-Image-Base's transformer on CUDA, through tf-moss-image, on an RTX 5090 and a GB10.
- Every step's cosine against the publisher's torch bf16 pipeline is 0.99967 or better, and a 1024x1024 picture is
44.0 dB (RTX 5090) and 45.9 dB (GB10) PSNR from torch's.
- 50 guided steps at 1024x1024 take 24.6 s on an RTX 5090 and 69.6 s on a GB10, 1.26x and 2.03x faster than torch.
- On CUDA, the prompt encoder and image decoder still run in torch, and nothing is served yet. See the
recipe (#1528, for #1518).
- DeepSeek-V4.1's release checkpoint (deepseek-ai/DeepSeek-V4.1-Flash-1017) is read as published: FP8 blocks,
MXFP4 routed experts and the FP8 Engram table, beside the EXL3 trellises. tf-dsv41-engram-pack writes the Engram
tables as row shards with a checked manifest. The CUDA engine runs EXL3, so it refuses the release checkpoint by name
before loading (#1705, for #1676). On Macs, its FP8 block-scaled rows and MXFP4 experts have their own Metal
kernels under the kernel contract, with Kimi's left as they are, and its routed experts a grouped prompt GEMM; nothing serves through them yet (#1706, #1707, #1708).
- LFM3-1.7B and LFM3-VL-2B: their official MLX 4-, 6- and 8-bit checkpoints are admitted and checked ahead of
their engines (#1513, part of #1491).
- XPU on Gen12LP (Iris Xe, DG1): Nemotron's kernels build, with a plain bf16 fallback for its two DPAS products. DPAS
builds are unchanged (#1673).
- Qwen3.8-27B on two Macs, groundwork: the 27B family can load one rank's half of the checkpoint and run its
layers with the pair's row exchanges, with one Mac's sums. Nothing serves through it yet, and on one Mac replies are
the same tokens (#1433, after #1432).
Speed
- MLX 4-bit decode on GB10s: projections that read the same input run as one grouped launch, each output
byte-equal to its separate launch, for the dense targets, Nemotron, the DFlash2 drafter, Gemma 4 and Spark.
- Nemotron 3.5 Lightning: +13.3% at 8 streams, +5.9% at 2.
- Qwen3.8-27B with DFlash2: +4.5% at 1 stream, +3.5% at 4, and +3.7% to +5.0% decoding after a prompt (#1783).
- Shared draft rounds price each stream's drafts first and skip proposal work that would get no verify rows, for
Nemotron, Qwen's MTP, GLM, Gemma 4 and attached LFM2 drafters, on CUDA and Metal. Nemotron 3.5 Lightning with MTP:
+9.0% greedy and +4.7% sampled at 16 streams on an M5 Ultra; +2.1% at 16 and +1.1% at 8 on a GB10 (#1779).
- Gemma 4 E4B on Intel GPUs: prompts read 2.0 to 2.6 times faster on an Arc Pro B70, and decode is 4% to 8% faster
at 32K to 64K tokens, with the same logits (#1746, part of #1222).
- HIP prompt attention on gfx1151: cold first token is 1% to 5% faster from 2K to 64K tokens on Qwen3.8-27B, with
the same output bits. The tile is now under the kernel contract (#1788).
- HIP prompt projections on gfx1151 round to the activation type in their tile instead of writing fp32 and
launching a cast, with the same bytes. Cold prompts on Qwen3.8-27B are 2% to 3% faster (#1685).
- Qwen3.8-27B on M5 Ultra: a prompt chunk of more than 128 rows reads each 4-bit weight matrix once per chunk
(#1738).
- Qwen3.8-27B on Metal with DFlash2: the drafter reads its sliding-window keys and values in place while the
context hasn't wrapped, instead of copying them each round (#1727).
- Flash Next's MTP head norm: 1.4x to 2.5x faster on a GB10 and 4x to 5x on a Radeon AI PRO R9700, at 1 to 16
rows, with the same bytes (#1702, fixes #1689).
- EXL3 MoE models on sm120 and sm121 (GLM-5.3-Flash and the Qwen3.5-MoE EXL3 packs): the grouped experts kernel
loads a warp's next tile while it decodes the current one, with the same output bits. 11% faster at
MiMo-V2.6-Flash's expert shapes on a GB10. GLM-5.3-Flash on two GB10s is within 1% of its previous speed (#1505).
- DeepSeek-V4.1 on CUDA: each window backs up only the ratio-2 recurrent carries, not every layer's sliding ring
(89 device copies, 6.4 MB a window on Flash-1017) (#1752). DSpark draft passes no longer wait for every side stream
(#1751).
- Flash Next on CUDA, opt-in: TFFLASHNEXTCHUNK_SOLO (up to 16,384 rows) lets a lone prompt take a larger chunk.
At 4,096 rows on a GB10, its first token comes 5% to 8% sooner. It costs 2.25 GiB more scratch, and a request that
arrives during a large pass can wait up to about 2 s (#1733, option C of #1554).
- GLM-5.3-Flash on CUDA, opt-in: TFGLMCHUNK_SOLO=4096, set on every rank, lets a prompt pass take 4,096 rows
while no other stream is decoding, with the same bytes. On GB10s a cold prompt's first token comes 6% to 9% sooner
from 8K to 128K tokens. It holds 1.6 to 1
TensorFold 1.2.02026-10-31
TensorFold 1.2.0
1.2.0 makes Qwen3.8-27B faster on Macs. On an M3 Ultra with DFlash2 it decodes 14% faster than 1.1.9 at one stream
and 44% faster at 16; on an M5 Ultra, 6% faster at one stream and 25% at 16. On a GB10, none of the ten models we
read got slower. It adds a new family,
MiniMax-M3.1-Flash on one 256 GB M5 Ultra, and new checkpoints and hardware: NVIDIA's NVFP4 Qwen3.6-35B-A3B, dense
EXL3 Qwen3.8-27B on CUDA, and Gemma 4 E4B on Intel's Arc Pro B70. Every speedup keeps replies token-equal to 1.1.9.
One change is deliberate: pictures on the Qwen-VL models are now prepared as the checkpoints' own processor prepares
them, so replies to pictures can differ slightly from 1.1.9. Text-only replies are unchanged. The full list is in
the changelog.
Measured
Every row was measured with the same tokens as its base.
| What | Hardware | Base | Before | After | Change |
|---|---|---|---|---|---:|
| Qwen3.8-27B MLX 4-bit, DFlash2, 1 stream | M3 Ultra | 1.1.9 | 80.0 tok/s | 91.6 tok/s | +14.4% |
| The same, 4 streams | M3 Ultra | 1.1.9 | 127.0 | 150.3 | +18.3% |
| The same, 16 streams | M3 Ultra | 1.1.9 | 145.2 | 209.4 | +44.2% |
| The same, prefill | M3 Ultra | 1.1.9 | | | +17% |
| Qwen3.8-27B MLX 4-bit, DFlash2, 1 stream | M5 Ultra | 1.1.9 | 168.5 tok/s | 178.3 tok/s | +5.8% |
| The same, 4 streams | M5 Ultra | 1.1.9 | 298.0 | 334.6 | +12.3% |
| The same, 16 streams | M5 Ultra | 1.1.9 | 378.5 | 474.0 | +25.2% |
| Qwen3.8-27B, one stream, code / chat (#1726) | M5 Ultra | main before it | | | +2.6% / +4.6% |
| Qwen3.8-27B, four greedy code streams (#1726) | M5 Ultra | main before it | | | +18.6% |
| Flash Next, sampled code / chat at 5 streams, 5 x 131,072 (#1609) | GB10 | main before it | | | +24% / +19% |
| The same at 4 streams | GB10 | main before it | | | +15% / +19% |
| Qwen3.6-35B-A3B MTP, 2 / 4 / 8 users (#1562) | GB10 | streams drafting one at a time | 116 / 138 / 159 tok/s | 137 / 182 / 220 | |
| LFM2.5-24B-A2B, a 2,048-row prompt chunk (#1590) | M1 Pro | main before it | 243 s | 3.9 s | |
| LFM2.5-24B-A2B, 16 greedy streams together (#1590) | M1 Pro | main before it | 11 to 17 tok/s | 116 to 123 | |
| Qwen3.8-27B MLX 4-bit, cold prefill at 2k (#1663) | Ryzen AI Max+ 395 (gfx1151) | main before it | 8.60 s | 5.83 s | 32% faster |
| Qwen3.8-27B, cold prompt at 2k (#1648) | gfx1151 | main before it | 8.64 s | 7.60 s | |
| GLM-5.3-Flash XSFP4 load at TP=2 (#1616, #1636) | two GB10s | before #1616 | 327 s | 51 s | |
| A 2,000-message DeepSeek-V4.1 chat's peak memory while its template renders (#1728) | any | main before it | 154 MB | 15 MB | |
New models and hardware
- MiniMax-M3.1-Flash on Metal (new family): the minimax_m3 family serves MiniMax's 428B-A23B model on one 256 GB
M5 Ultra. It supports tools, structured output, thinking on or off, images, the prompt cache, two streams, and
DFlash2 or DSpark drafts that keep replies equal to plain decoding.
- It reads two community checkpoints. Both are lossy conversions of MiniMax's NVFP4 release, labelled with their
cost against it: mean KL 0.244 on mix34 and 0.295 on mix256k. Everything from those weights on is exact, with a
bf16 key/value cache (147,456 tokens on mix256k, 32,768 on mix34).
- With DFlash2, one stream decodes 44 to 58 tokens a second on mix256k.
- MINIMAXLOADLIMITGB sets the memory budget, and MINIMAXDRAFTER picks the drafter (#1235, for #1519).
- NVIDIA's NVFP4 Qwen3.6-35B-A3B on CUDA (nvidia/Qwen3.6-35B-A3B-NVFP4) serves as downloaded, and --drafter
naming the same checkpoint drafts with its MTP layer. One GB10 decodes 66 tok/s without drafts and 112 to 125 with
them. Drafted, concurrent and resumed replies equal their plain runs (#1662).
- Dense EXL3 Qwen3.8-27B on CUDA: turboderp's dense EXL3 checkpoints now load instead of being refused, and
DFlash2 drafts them. At 4.00 bpw on a GB10, decode runs at 45.8 tok/s for chat and 71.9 for code with DFlash2,
against 15.3 and 15.5 without (#1465). The drafter reads only the two vocabulary spans it proposes from, about
545 MB fewer reads a round at a 6-bit head (#1507).
- ModelOpt Qwen3.8-27B on sm80 to sm89: NVFP4 and FP8 checkpoints now serve on cards such as a Jetson AGX Orin.
This path hasn't run on such a card yet. Nothing changes on sm_90 and newer (#1470).
- Gemma 4 E4B on Intel's Arc Pro B70 (Xe2): the XPU backend serves mlx_lm.convert 4-bit checkpoints, with an
optional 8-bit attention, up to 8 streams with shared forwards, and the prompt cache. On the B70, 2,048-token prompts
read at 6,772 tok/s, one stream decodes at 135.6 tok/s, and 4 streams reach 426 tok/s together, each reply equal to
that stream's reply alone. The audio tower and the drafter aren't in yet (#1552, for #1222).
- Gemma 4 structured output on CUDA: responseformat and toolchoice now constrain its replies, with or without
a drafter (#1386).
- Three GB10s in a ring without a switch now connect at three ranks or more. An 8 KiB exchange takes 10.7 µs,
against 26.3 µs for NCCL. A ring of four still falls back to NCCL (#1516).
- gfx1100 with clang 21 (opt-in): -Dhip-relax-branches=true builds the dot2 prompt tiles with Ubuntu's clang 21.
It hasn't run on a gfx1100 GPU yet (#1542).
Speed
Each of these keeps replies token-equal to main before it.
- Metal:
- Qwen3.8-27B:
- The quantized tiles unroll their fixed loops, with bit-identical outputs, and a shared round whose drafts get
no rows skips the drafter (#1686, for #1621).
- On M5 Macs, one stream takes the lane loop by default (#1726, for #1621). On an M3 Ultra, the same change makes
16 streams 3.5% faster and one stream 1.9%.
- Nemotron 3.5:
- On M3-class GPUs, the decode attention and the experts' 2-row tiles unroll their fixed loops. On an M3 Ultra,
Lightning decodes 1.8% faster at one stream (#1704, for #1621).
- On M1-family Macs, routed experts in windows of 16 rows or more take two members a pass. On an M1 Max the
kernels take 14 to 20% less time at 64 to 128 rows (#1466).
- LFM2.5-24B-A2B on M1 and M2 Macs: the prompt tiles store their matrix fragments whole, and run at 2.4 to
2.7 TFLOPS on an M1 Pro. M3 and newer compile them as before. The M2 takes the M1's path but wasn't measured
(#1590, fixes #1523).
- CUDA:
- Qwen3.6-35B-A3B: every stream of a shared round drafts together in the MTP layer (#1562).
- Flash Next: a shared round's sampled streams draft in the batched MTP head pass. Greedy streams and time to
first token are unchanged (#1609).
- EXL3:
- A context copy beside the dense 27B now needs 20 matching tokens instead of 5. On an RTX 3090 Ti, decode after
long prompts is up 12.8% sampled and 18.5% greedy (#1477).
- The dense projections zero their split-K counters only when the buffer grows. On a GB10 with the 27B 3.50 bpw
pack, decode is 0.4% to 0.9% faster (#1475).
- The trellis loads skip L1. On three GB10s at TP=3 with GLM-5.3-Flash EXL3 4 bpw, decode rose 2.3% on one stream
(#1534).
- GLM-5.3-Flash load: reader threads fetch each layer ahead (#1616), and 8 threads decode and pack each layer's
routed experts (#1636). On three GB10s it now loads in 46 s, from 268 s.
- Nemotron's router top-k runs on the shared MoE top-k: 2.74 µs a call instead of 4.37 on a GB10 (#1354).
- HIP:
- gfx11's matrix-core prompt tile no longer spills to scratch (#1663) and runs in bands of four row blocks
(#1648). Cold prefill on a gfx1151 at 64k is 24.5% faster from #1663. #1648 cuts 365.7 s to 330.8 s.
- RDNA4 short routed prompts: on an R9700, cold prefill takes 0.53 to 0.80 times as long on Qwen3.6-35B-A3B at
33 to 128 tokens. Other GPUs are unchanged (#1607).
Opt-in
- GLM --mtp-confidence: a draft chain stops where its chance falls under the flo
TensorFold 1.1.92026-10-29
TensorFold 1.1.9
1.1.9 speeds up prompts on DGX Spark and Flash Next on Macs: on a GB10, Qwen3.8-27B reads prompts 9% to 11% faster than 1.1.8 and Kolibri-1 3% to 8% faster, and on an M3 Ultra, Flash Next's copy-heavy edits run at 212 to 257 tok/s and sampled prose at up to 136 tok/s, from 122.5. On an M5 Ultra, Qwen3.8-27B reads prompts about 4% faster. Every model gives the same tokens as 1.1.8. The full list is in the changelog.
Speed
- Qwen3.8-27B prompts on CUDA: a prompt runs up to four of its projections in one launch, each keeping the shipped BF16 kernel's block order, and takes a 2x2-warp tile at large prompt widths (#1439, for #1398). Outputs are the same bytes. On a GB10 with the MLX 4-bit checkpoint, prefill is 10.8% faster at 2k tokens, 9.7% at 8k and 7.8% at 32k. Prompts on the default path (without TFQWENKV_PAGES) also take whole-window attention kernels again, which wins back the long-prompt prefill 1.1.7 lost (#1458): 0.5% at 32k and 0.8% at 64k.
- Kolibri-1 on CUDA, and Drama 3 with prompt chunks of more than one row: the block-FP8 routed experts' prompt kernels copy weights into shared memory in stored order, so each 32-byte sector is read from L2 once instead of twice (#1460). Outputs are the same bytes. On a GB10, Kolibri-1's prompts ran 13% faster at 2k tokens and 3% to 4% faster from 8k to 64k.
- Flash Next copy drafts on Metal: when a reply repeats text from its prompt or from earlier in the reply (edits, rewrites, renames, quotes), its copied drafts verify in windows as wide as their measured acceptance pays for, about 11 rows where they stopped at about 4. Requests whose prompt plus maxtokens passes about 2,000 tokens, which includes every request at the default maxtokens of 4,096, now get the GPU gate that sets each round's width, as shorter requests already did. On an M3 Ultra, copy-heavy edits ran 212 to 257 tok/s and rewrites 222 to 300; at the default max_tokens, prose ran 138 to 142 and long documents 173 to 179.
- Sampled Flash Next replies on Metal: the checkpoint samples by default (temperature 1.0, topk 20, topp 0.95), and a sampled reply now gets the GPU gate, MTP drafts drawn under the reply's own keyed rule, and a faster draw: for topk up to 64 the GPU keeps each of 32 slices' topk and merges them in a second pass, about 29 µs where the whole-vocabulary draw took about 700 µs, with the same sums, cuts and tie rule. It picks the same token at every seed and position. On an M3 Ultra, sampled prose ran 122.5 to 136.0 tok/s, code 180 to 194 and hard prompts 133 to 146. Greedy replies are unchanged.
- Qwen3.8-27B prompt rows on M5: a request asks for 2,048 prompt rows when no other request is decoding, and caps each chunk at 512 beside a decoder (#1478). The larger frame is used only when the context, every stream and the full prompt scratch fit beside the weights and the existing 8 GiB margin, and startup reserves that scratch. Earlier chips keep 512 rows; TFPROMPTROWS overrides the width. On an M5 Ultra, prefill is 3.7% to 4.5% faster from 2k to 64k tokens.
- Nemotron 3.5 on Metal: a GPU round prices its draft head's depth from the backend's window-cost table, and no longer steps back up to a level that has shown it doesn't land (#1115). With Nemotron 3.5 Lightning 30B-A3B (MLX 4-bit), decode after 2k to 64k-token prompts is 1.2% to 2.4% faster on an M3 Ultra and within 0.3% on an M5 Ultra.
- Qwen3.8-27B follow-up turns on Metal: a follow-up turn resumes from the state its conversation's last reply left in the server's first slot when the prompt cache holds nothing as far, so on a Mac whose cache can't keep a long conversation (a 32 GB Mac) the next turn reads only its new tokens instead of its whole prompt (#1197). Replies are the same tokens as a fresh prompt pass. TFQWEN27SLOT_RESUME=0 turns it off.
- Qwen3.5-2B conversations on Metal (opt-in): with --prompt-cache-gib N, a request that extends a conversation resumes from its deepest kept prompt state in RAM instead of prefilling it again (#505). On an M5 Ultra the next turn of a 28.5k-token chat resumed with 28,544 tokens cached, and on an M5 Ultra and an M3 Ultra every resumed reply equalled a fresh one. Without the flag nothing changes.
- Flash Next on CUDA with NVFP4 experts: at decode, the routed experts' work goes out in equal waves instead of leaving a short last wave (#1351). Each output keeps the same multiply-adds in the same order. On a GB10, served decode is 0.6% faster at one stream and 0.2% at two.
- GLM-5.3-Flash on CUDA:
- At long positions a decode step's DSA select gives each row a 1,024-thread block where it used 128, with the same picks in the same order (#1424). On three GB10s at TP=3, decode after a 136k-token prompt ran 1.8% faster.
- With two or more ranks, a prompt chunk's DSA indexer splits across the ranks from position 2,048 on, and an all-gather gives every rank every row's picks (#1404). Replies are the same tokens, and decode never splits. TFGLMINDEX_SPLIT sets the starting position, and 0 turns it off. On two GB10s at TP=2, prefill at 128k tokens was 1% to 2% faster.
- With TENSORFOLDPREFILLSHARE below 1, a long prompt's step ends on the GPU before it returns when the lane host bills it to decoding streams, so its tail no longer runs in the decoders' next round (#1352). At the default of 1 nothing changes.
Server
- The serving line names its drafters: for example drafts: copies only, drafts: MTP + copies or drafts: DFlash2 (z-lab/Qwen3.8-27B-DFlash2) + copies, where it used to say drafts: on whether or not a drafter was loaded (#1515). Drafting itself doesn't change.
- Qwen3.8-27B profiling on Metal (opt-in): TFQ27PROFILE=N logs each kernel class's GPU time, the DFlash2 drafter's included, every N decode rounds, and adds gpu_seconds to replies (#1215). It is a diagnostic: on an M5 Ultra one stream decodes at about a ninth of its usual rate with it on. Without it replies are byte-identical to before.
Fixes
- --compact-at auto on small windows: on a context window smaller than its 16,384-token reserve, it reserves fifteen percent of the window instead, so a short request on --context 8192 is no longer compacted (#1536). Larger windows keep the 16,384-token reserve.
- Release builds: the release path check no longer refuses a clean binary whose address tables happen to hold bytes that read as \.o (#1533). Every other rule is unchanged.
HIP
- AMD RDNA4 cards (gfx1201) take a tuning table of their own, which keeps gfx1100's bounds for now, so no kernel choice changes; zig build hip-affine-tune -Dhip=true times the MLX affine prompt tiles on a card to fill it (#1527).
Tests and tools
- CUDA kernels take their shared helpers from one set of headers; every kernel compiles to the same code as before (#1440).
- kernel-contract-limits no longer reports Qwen's prompt partials as launched past their bounds on every SM.
Release testing
1.1.9 against 1.1.8 on the same machines and checkpoints, alternating the two builds (1.1.8, 1.1.9, 1.1.9, 1.1.8). The cells were cold prompts from 2k to 64k, and the stream sweep at 1 to 8 streams (16 for LFM2) with every stream checked against its run alone. Every change above was also measured against main on its own before it landed.
- Every model gives the same tokens on both releases in every cell.
- DGX Spark (GB10):
- Qwen3.8-27B: prefill +11.1% at 2k, +10.4% at 8k and +8.8% at 32k. Decode and the sweep are within 0.2%.
- Kolibri-1: prefill +8.1% at 2k, +3.7% at 8k, +3.4% at 32k and +3.0% at 64k. Decode after the prompt is within 2%, about as much as the 1.1.8 runs vary between themselves. Across twelve alternating runs, one stream is within 0.4%.
- Qwen3.8 Flash Next FP4-Local: prefill +0.3% to +1.4%, decode +0.1% to +0.5%, and the sweep +0.8% to +0.9%.
- Nemotr
