oMLX

大语言模型 23.3k 2.1k Python 上架 2026-11-08 访问官网 ↗ 在 GitHub 打开

LLM 推理，为你的 Mac 优化在 Apple Silicon 上支持文本 LLM、视觉语言模型（VLM）、OCR 模型、嵌入模型和重排序模型。连续批处理和分层 KV 缓存，直接从菜单栏管理。 o...

#LLM（大语言模型）

项目信息

所有者: jundot
Star: 23.3k
Fork: 2.1k
开源协议: —
主要语言: Python
发布时间: 2026-02-13
最近更新: 2026-11-04

项目更新

0.7.32026-10-31
oMLX 0.7.3 主打三点：DeepSeek V4.1 提速、Ask AI 能力扩展、高并发长上下文省内存。 M5 上开启 INT8 Activation Prefill 后，64K 提示预填充从 2,882 升到 3,507 tok/s，195K 场景提升 29%；批量超过 8 条请求继续走融合解码内核，12 并发聚合解码快 123%。 Ask AI 新增语音对话：在 App > Ask AI 打开 Voice，按 Option-Shift-Space 开口，回答带字幕朗读；还能联网搜索、读取链接、查看你的屏幕（#4976），并提供兼容 OpenAI Realtime API 子集的 /v1/realtime 端点。多条长对话并行时，批量 KV 缓存改为原地更新而非复制，峰值内存最多降 35%，解码同步加快——16 请求的 Qwen3.6-27B 快 51%，共享 6.5K 系统提示的场景首字延迟降 72%。模型管理器同时支持增量更新，只拉取 Hugging Face 上变动的文件。
0.7.3rc12026-10-27
oMLX 0.7.3rc1 候选版发布，正式版可继续用 0.7.2。 Ask AI 是这版的主角：在任意应用里按 Option-Space 呼出问答条，悬浮在当前窗口之上却不抢焦点，回答以 Markdown 流式输出。斜杠命令可按预设指定模型与系统提示，例如 /translate 用小模型跑固定提示词并自动复制结果；Vision 模型支持 Command-V 贴图提问。入口在 App > Ask AI。 Model Settings > Advanced 新增 GPU 功率限制，范围 2–1000 W，仅在受控请求生成时生效，最后一个请求结束 15 秒后自动解除，基准测试不受影响，也不需要 root 或辅助程序。实测 M5 Max 上把功耗压到 5 W，27B 模型解码速度降 48%，但每 token 能耗省 52%——用速度换续航，适合电池办公。模型侧提速明显：Ling 3.1 Flash 改为直接加载原始 FP8 权重，Ling Flash 3.0/3.1 的 nextn 层以 Lightning MTP 运行（oQ 的 Preserve MTP 会保留），M3 Ultra 上 4K 上下文解码从 18.8 提到 34.3 tok/s，涨幅 82%。此外 Qwen3.8-Flash-Next oQ4e 解码 +24%，GLM-5.3-Flash 在 M5 之前的 Mac 上 +12%，Flint oQ3e 配合 TurboQuant 快至 3.2 倍，DeepSeek-V4.1-Flash 128K 预填充 +14%。
0.7.3.dev22026-10-25
oMLX 0.7.3.dev2 是开发版热修，修掉了给 MiniMax M3.1 开启 Lightning MTP 报错的问题（#4827），并清理了它非流式回复开头多余的字符，同时修复 DeepSeek V4 Flash oQ3.5e 的预填充。性能方面：MiniMax M3.1 Flash 支持 NVFP4 原始权重与 oQ 构建，开启 Lightning MTP 后单请求吞吐从 22.5 提到 42.0 tok/s，配合融合解码内核，短上下文再快 14%；M1、M5 机型分别针对预填充做了专属优化，M1 Ultra 预填充提速约 50%；DFlash 草稿模型改跑 fp16，GLM-5.3 批量解码也省下了 3.27 GB 内存。低比特 oQ 重新分配了位宽：输出头与嵌入层不再固定占 8 位，把位数让给注意力与 MLP 层，Qwen3.8-27B oQ2.5e 的 chat KL 降了 73%，MMLU-ProX 从 62.0% 升到 70.0%，模型体积几乎不变。
0.7.3.dev12026-10-24
oMLX 0.7.3.dev1（已由热修版 0.7.3.dev2 取代，后者修复了为 MiniMax M3.1 开启 Lightning MTP 时的报错）。本版新增 MiniMax M3.1 Flash：内置 DSpark 草稿模型以 Lightning MTP 运行，一次前向验证整批，开关前后输出完全一致；单请求吞吐从 22.5 提到 42.0 tok/s，再叠加融合解码内核，5K–64K 上下文还能快 21–23%。M1、M5 Mac 各自拿到专属路径，M1 预填充提速约 50%，M5 上 DeepSeek V4 的注意力改走张量单元；fp16 模型的 DFlash 草稿不再整段跑 fp32，GLM-5.3 解码更快也释放出 3.27 GB 显存。低比特 oQ 重新分配位数：输出头降为基值 +3 bit，注意力与 MLP 层优先升到 3 bit，体积波动不超过 0.07 GB，Qwen3.8-27B oQ2.5e 的 KL 因此降了 73%，评测分数同步走高。
0.7.22026-10-21
oMLX 0.7.2 oMLX 0.7.2 brings faster DeepSeek V4.1 on every oQ build, Lightning MTP across Macs in distributed inference, and Qwen3.8-Flash-Next GGUFs on Splash. It's only been a day since 0.7.1, but these were worth shipping now. Download: macOS 26 / 27 | macOS 15 Sequoia Faster DeepSeek V4.1 on oQ2e to oQ4e The fused decode kernels in 0.7.1 only read the MXFP weights of oQ4e, so oQ2e, oQ2.5e and oQ3e still decoded on a slower generic path. They now run on the fused kernels too, Lightning MTP verify shares work across the checked tokens, and prefill runs in 8192-token chunks. #4722 > DeepSeek-V4.1-Flash-1017 -mtp checkpoints, 16K prompt, M3 Ultra 512GB. | Quants | Decode, MTP off (tok/s) | Decode, MTP + CED (tok/s) | Prefill, MTP + CED (tok/s) | |---|---:|---:|---:| | oQ2e | 23.0 -> 39.5 (+72%) | 55.6 -> 59.2 (+6%) | 741 -> 1008 (+36%) | | oQ2.5e | 22.8 -> 39.5 (+73%) | 47.4 -> 67.1 (+42%) | 737 -> 1003 (+36%) | | oQ3e | 22.9 -> 42.7 (+86%) | 49.4 -> 68.2 (+38%) | 700 -> 949 (+36%) | | oQ4e | 32.8 -> 38.7 (+18%) | 53.8 -> 64.1 (+19%) | 773 -> 1010 (+31%) | With Lightning MTP + CED, prefill holds around 900 tok/s all the way to a 128K prompt. Lightning MTP in Distributed Inference Distributed inference now runs Lightning MTP on DeepSeek V4.1 and GLM-5.3 with pipeline or tensor splits. It turns on by the same rule as on one Mac, when the checkpoint has a draft head and the model setting does not turn it off. #4732 > These runs used a TCP ring over Thunderbolt, not RDMA, so the absolute numbers are not the best case. Read them for the gain from Lightning MTP. | Two M3 Ultra over Thunderbolt | Split | Sampled (tok/s) | Greedy (tok/s) | |---|---|---:|---:| | DeepSeek V4.1 Flash oQ4e | Pipeline | 28.7 -> 32.4 (+13%) | 29.1 -> 34.3 (+18%) | | DeepSeek V4.1 Flash oQ4e | Tensor | 13.3 -> 24.3 (+83%) | 13.5 -> 25.4 (+89%) | | GLM-5.3 Flash oQ4e | Pipeline | 37.0 -> 42.6 (+15%) | 37.6 -> 47.4 (+26%) | | GLM-5.3 Flash oQ4e | Tensor | 13.7 -> 27.7 (+102%) | 13.7 -> 25.7 (+88%) | Decode with Lightning MTP off -> on, 512 generated tokens, measured before the #4722 kernels. Sampled is temperature 1.0, topp 0.95, topk 20. - No more pipeline stall: an idle pipeline stage let the GPU slow down, which held DeepSeek V4.1 at about 1 tok/s across two Macs. It now decodes at about 28 tok/s with Lightning MTP off. - Thunderbolt warnings: the plan preview warns when a Thunderbolt Bridge has only a self-assigned 169.254 address, which made the cluster fall back to the LAN or Wi-Fi without a word, and when the link runs over Wi-Fi. More Models on Splash With Splash 1.4.0, Qwen3.8-Flash-Next GGUFs run on Splash, and oMLX offers the matching incoai/Qwen3.8-Flash-Next-DFlash2 draft when it is missing. Splash 1.4.0 streams experts from SSD when memory is tight, and the Q2_0 build ran on a 64 GB M5 Pro. By @arnavprabhu in #4719, with a follow-up in 4f605d6d8. - Split GGUFs: multi-part GGUFs at the repo root or in variant folders like Q2_0/ download, list and delete as one model, together with the shared vision projector. - OptiQ checkpoints: mlx-community OptiQ builds that keep their vision weights in a subfolder are staged the way Splash reads them, so they start instead of failing with a 409 (#4718). Other Changes - Claude Code: notes that Claude Code sends after a tool result (hook context, loaded rules, reminders) stay in place instead of moving into the leading system block, so a turn no longer prefills the whole conversation again (@JarJarFran in #4712). Splash keeps later system messages in place too (@phaedrus1992 in #4708) and receives preserve_thinking (@isaac-cf-wong in #4594). - Claude Code auto mode: classifier requests get a 256-token thinking budget, so reasoning models answer before the classifier deadline (f0e3f6a84, with @phaedrus1992 from #4714). - DFlash: ParoQuant Qwen3, Qwen3-MoE and Qwen3.5 targets can use DFlash (87251ade6, with @popfido from #4685). - SpecPrefill: a SpecPrefill request no longer shifts the RoPE positions of requests already decoding beside it, and sliding-window models no longer process the last prompt token twice (@billythegod in #4688). Gemma 3 global attention layers get the right rotation in sparse prefill (78d93f385). - API: omlx launch works with an IPv6 bind host, and zero or negative values for max tokens, rerank top_n, embedding dimensions and TTS sampling fields are rejected (@billythegod in #4689, bed3f3520). - Attachments: browser files sent as application/octet-stream are converted by their extension (@saichowdary007 in #4179). - Benchmarks: throughput tg rows decode the full length instead of stopping at the model's end of turn (@zviratko in #4687). - Cluster: "Forget this Mac" no longer fails with a 500 (@bercedev in #4716). - Web UI: the navbar menus and hover-only copy and unload buttons can be reached from the keyboard (#4729), console text meets WCAG AA and links use the grey scale (#4727), the dark theme shows the memory meter, plain commands and off switches (#4728), Status lines up with the other tabs (#4731), status card text stays inside narrow cards (#4706), and the log line count and hot cache unit read correctly (#4707), all by @LXD-8. The SpecPrefill link in model settings works again (@shaanchander in #4723), and Chinese bench metric cards lead with the abbreviation (@LXD-8 in #4699). - Homebrew: formula updates no longer race when releases ship back to back (@billythegod in #4691). Upgrade Notes - Source installs: run make install (or make kernels) after updating. The sparse attention kernel changed in #4722. - DeepSeek V4.1: prefill now runs in 8192-token chunks, so peak memory at a 16K prompt is 4-7 GB higher. With the SSD cache on, cache blocks are 8192 tokens, and a request that changes something inside the last 8192 tokens recomputes from that boundary. - Cluster: Lightning MTP now turns on for deployments whose checkpoint has a draft head. Turn it off in Model Settings to keep the old behavior. - Benchmarks: tg rows now decode the full length, so gen tok/s is not directly comparable with earlier runs of models that used to stop early. New Contributors @phaedrus1992 in #4708, @JarJarFran in #4712, @bercedev in #4716, @shaanchander in #4723. Full changelog
0.7.12026-10-20
oMLX 0.7.1 正式版发布。模型下载器现在支持通过 Splash 获取 GGUF：可按 MLX/GGUF 过滤仓库并自行勾选文件，GGUF 变体会连同视觉投影器一并下载，Qwen3.8-27B、Qwen3.6-35B-A3B 以及 Splash 1.3.2 下的 NVFP4、FP8 检查点都能在 Splash 上运行；内存吃紧时改为自动收缩上下文，不再直接返回 409 拒绝启动。 DeepSeek V4.1 Flash 解码改用融合内核，多请求改为每步只读一次权重。M3 Ultra 上单请求解码从 20.9 提升到 34.3 tok/s（+64%），8 并发合计吞吐从 19.4 冲到 91.8 tok/s（+373%），输出与不开 MTP 时逐 token 一致。新增 Affine4、Affine8 两种 KV cache 压缩格式，全注意力 KV 占用最多减少 74%，预填充与解码速度基本无损，可在 Model Settings > Memory Saving 中切换。多个模型同时运行时 GPU 卡死、被 macOS 重置的问题也修掉了：所有引擎改走共享命令队列，七个模型压测中 GPU 重置从 5 次降为 0，失败请求归零。分布式推理覆盖面更广，GLM-5.3 与 DeepSeek V4.1 支持跨 Mac 的流水线或张量并行，视觉模型权重也可当作纯文本模型部署。
0.7.1rc12026-10-15
oMLX 0.7.1rc1 候选版带来了 Muse Spark Open：Meta 这款 816B MoE（激活参数 42B）支持文本与图像输入、工具调用以及 low 至 xhigh 推理档位，BF16 与 NVFP4 权重可直接加载；oQ3e 构建（370 GB）能在 512 GB 的 Mac 上运行，M3 Ultra 实测预填充 225 tok/s、解码 21.3 tok/s。模型下载器新增“推荐”标签，按你的 Mac 实际跑得动的规格筛选 oQ 模型，从最合适到更大尺寸依次排列，并标注解锁各自所需的设置；带 Lightning MTP 的模型默认开启，下载完即提速。Qwen3.8 与 Qwen3.6 现在可交给外部引擎 Splash（DFlash2 投机解码）在独立进程服务，模型仍保留在 oMLX 的列表、API、仪表盘与基准测试中，在模型设置 > 高级打开“使用外部引擎”即可。界面同步重做：Web 端模型设置分组并配侧边菜单，每项一行带简短说明；macOS 端改为系统设置式的分页、即改
0.7.1.dev22026-10-15
oMLX 0.7.1.dev2 是开发版本，求稳可继续用 0.7.0。这版四块更新：模型下载页新增「推荐」标签，按你的 Mac 实际跑得动的内存列出可加载的 oQ 模型，从最合适排到更大规格，并标出解锁各自的设置；带 Lightning MTP 的模型默认开启，下载完就能提速。Qwen3.8、Qwen3.6 现可交给外部引擎 Splash 服务，独立进程运行并支持 DFlash2 投机解码，在模型设置的「高级」里勾选即可，模型仍留在列表、API 与统计中，视觉模型也能收图像。Web 端与 macOS 端的模型设置改成分组配侧边菜单、边改边存，聊天页把模型、配置档和推理档位收进输入区，滑块只给当前模型支持的档位，日志改为表格视图。DFlash 也已接入批量引擎。
0.7.1.dev12026-10-09
oMLX 0.7.1.dev1 为开发版，更新集中在三块。新增决策模型支持：POST /v1/systemone 可调用 Clef、Clef-Flash 与 OpenJev，读一次状态即返回各选项概率，不再生成文本，模型可自动识别，也可手动设为 decision。并发请求的预填充合并为一次前向计算，MoE 专家权重只读一遍，各请求缓存保持独立，首 token 延迟降低 8%~25%，覆盖 Qwen3.5/3.6/3.8、GLM-5.3 与 Hy3。另一批改动是纯提速：Qwen3.8-Flash-Next 关闭 MTP 后解码从 39 升到 91 tok/s，Qwen3.6-35B-A3B 提升 31%，GLM-5.3-Flash 提升 30%，DeepSeek V4.1 Flash
0.7.02026-09-30
oMLX 0.7.0 正式发布，这版在 0.7.0rc1 基础上修复了候选版暴露的回归问题。三处主要变化：Qwen3.8-Flash-Next 的预填充与解码双双提速，MoE 融合后路由专家的调度从五次降到两次，且 Lightning MTP 的验证行与串行解码逐位一致，开启它不再改变贪心输出；GLM-5.3-Flash 换上流水线预填充与 M5 张量单元上的稀疏 MLA 注意力，解码结果保持不变，DFlash2 同步受益，长上下文场景提升最大；内存保护机制则整体重写。MiMo V2 的推理速度也有改善。