v0.6.02026-07-02
- 亮点:
- 新增 chitu.run:单可执行文件即可启动多节点、多实例、PD 分离等复杂任务。
- 新增功能:
- 新模型:GLM-5.2。
- 昇腾 910B 平台支持 GLM-5 系列。
- 支持在请求中设置 reasoning_effort。
- 增加 DeepSeek-V4 工具调用支持。
- INT8 W8A8 MoE 增加 triton 算子实现。
- 支持 prefill 上下文并行(Prefill CP),目前支持 DeepSeek-V3.x、GLM-5.x。
- 性能优化:
- 优化 DeepSeek Sparse Attention 的 FP8 prefill 性能。
- 面向 DeepSeek-V4 的多处优化。
- 集成海光平台的若干算子。
- 面向海光平台优化多个辅助算子。
- 优化 DP 时的元信息通信。
- 增加 cudaroutegate_norm 融合算子。
- 增加 layer_norm 算子的 triton 实现。
- 问题修复:
- 使设置 infer.full_warmup 时对各分支情况的 warmup 更充分。
- 修复设置 infer.full_warmup 时的 KV cache 分配错误。
- 修复 MoE 串行回退实现的量化支持。
- 修复 MoE dispatcher 的 allgather 实现处理空的 DP rank 时的错误。
- 修复 infer.ppsize > infer.maxbatch_size 时的错误。
- 修复 PD 分离对长度超出限制的请求的错误处理。
- 修复导致推理死锁的若干问题。
- 修复启用 MTP 时 EP 通信 buffer 的潜在溢出问题。
- 修复 DeepSeek Sparse Attention 中 indexer 的潜在 int32 下标溢出问题。
- 持续迭代:
- 重构 native layout 机制。
- 利用 coordinator 分配赤兔内部使用的 TCP 端口。除 coordinator 本身外,其余内部端口不再需要用户指定。
- 将 nvidia-ml-py 从必选依赖改为可选依赖,并不再使用上游已弃用的用法。
------
- Highlight:
- New chitu.run: Single executable to launch complex jobs including multi-node, multi-instance, or PD-disaggregation.
- Features:
- New model: GLM-5.2.
- Supported GLM-5 series on Ascend 910B platform.
- Supported resoning_effort in requests.
- Supported tool calling for DeepSeek-V4.
- New triton implementation for INT8 W8A8 MoE.
- Supported Prefill Context Parallel (Prefill CP), supporting DeepSeek-V3.x, GLM-5.x.
- Optimizations:
- Optimized FP8 prefill of DeepSeek Sparse Attention.
- Multiple optimizations for DeepSeek-V4.
- Integrated multiple operators for Hygon platform.
- Optimized multiple auxiliary operators for Hygon platform.
- Optimized metadata communication for DP.
- New fused operator cudaroutegate_norm.
- New triton implementation for layer_norm.
- Bug fixes:
- More comprehensive warmup when infer.full_warmup is set.
- Bug fix on KV cache allocation when infer.full_warmup is set.
- Bug fix on quantization in serial fallback of MoE.
- Fixed the allgather implementation of MoE dispatcher for empty DP ranks.
- Fixed for infer.ppsize > infer.maxbatch_size.
- Fixed error handling for out-of-limit sequence length in PD disaggregation.
- Multiple fixes on dead locks.
- Fixed potential buffer overflow in EP communication when MTP is enabled.
- Fixed potential int32 index overflow in the indexer of DeepSeek Sparse Attention.
- Continuous iteration:
- Refactored native layout mechanism.
- Allocate internal TCP ports in Chitu with coordinator. Explicit set of all internal TCP ports is no longer needed except for the coordinator itself.
- Change nvidia-ml-py from a required dependency to an optional dependency, and drop deprecated usage of this package.
------
构建产物 1(推荐): chitu.run 可执行文件。见下方“Assets”。
Artifact 1 (recommanded): chitu.run executable. See "Assets" below.
------------------------
构建产物 2:docker 镜像:
Artifact 2: docker images:
- 英伟达 / NVIDIA (arch 8.0, 8.9): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-nvidiaarch80_89:v0.6.0
- 英伟达 / NVIDIA (arch 9.0): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-nvidiaarch90:v0.6.0
- 沐曦 / MetaX: qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-muxi:v0.6.0
- 昇腾 / Ascend (A2): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-ascend_a2:v0.6.0
v0.5.72026-06-04
- 利用融合算子替换 DeepSeek-V4 中的多处朴素实现。
- 优化 DeepSeek-V3.2 等模型中稀疏 attention 读取 indexer KV 的操作。
- 集成 hpc-ops 中的 attention 算子。
- 集成海光平台的若干 w8a8 算子。
- 减少算子分发逻辑中的 CPU 开销。
- 减少对请求的流量控制逻辑中的 CPU 开销。
- 减少 MoE prefill 时布局转换逻辑中的 CPU-GPU 同步开销。
- 允许请求中工具定义中的参数类型字段为非 str 类型。
- 赤兔启动时一律根据 Infiniband 设备情况自动设置环境变量,不再依赖 script/ 目录中的额外启动脚本设置。
- PD 分离兼容 MTP。
- PD 分离部署支持 Grafana 监控面板。
- 修复若干 KV cache 泄漏问题。
- 删除一处对于 MTP 过于严格的 assert。
- 修复 w8a8 MoE 的 triton 实现中的一处访存越界错误。
------
- Replaced multiple brute-force implementation in DeepSeek-V4 with fused operators.
- Optimized index KV reading in sprase attention in DeepSeek-V3.2 and similar models.
- Integrated attention operators from hpc-ops.
- Integrated multiple w8a8 operators on Hygon platforms.
- Reduced CPU overhead in operator dispatchment.
- Reduced CPU overhead in flow control of requests.
- Reduced CPU-GPU synchronization overhead in layout transformation during MoE prefill.
- Supported argument type field to have types other than str in tool definitions in requests.
- Infiniband-related environment variables are now always configured automatically when starting Chitu, and no longer rely on extra starting scripts in script/.
- PD-disaggregation is now compatible with MTP.
- PD-disaggregation is now compatible with Grafana metrics pannel.
- Fixed multiple KV cache leaks.
- Removed an overly strict assertion for MTP.
- Fixed an out-of-range memory access in the triton implementation of w8a8 MoE.
------
Official Docker images / 官方 docker 镜像:
- 英伟达 / NVIDIA (arch 8.0, 8.9): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-nvidiaarch80_89:v0.5.7
- 英伟达 / NVIDIA (arch 9.0): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-nvidiaarch90:v0.5.7
- 沐曦 / MetaX: qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-muxi:v0.5.7
- 昇腾 / Ascend (A2): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-ascend_a2:v0.5.7
v0.5.62026-05-21
- 新模型:Qwen3.6-27B、Qwen3.6-27B-FP8、Qwen3.6-35B-A3B、Qwen3.6-35B-A3B-FP8、DeepSeek-V4-Flash-Base(仅实验性)、DeepSeek-V4-Flash-FP8(仅实验性)。
- 选择模型时,所有“Qwen3_5”字样变更为“Qwen3.5“。
- 工具调用请求对可用工具的描述中,“required” 字段不再为必填项。
- 赤兔中内建的大部分 CUDA 算子支持海光平台。
- 集成海光平台的多个新算子。
- infer.fusesharedexperts 选项兼容 EP。
- 令 DeepSeek-V3.2 等模型中的 indexer cache 也能接受前缀缓存。
- GLM-5/5.1-FP8 兼容 infer.mla_absorb=absorb-without-precomp 选项。
- MTP 兼容 infer.schedule_overlap 优化。
- (TP×DP)+(ETP×EP) 混合并行。
- 减少 DeepEP + DeepGEMM Prefill 中的 CPU-GPU 同步。
- 面向前缀缓存命中率优化 DP rank 和多 instance 间的请求路由。
- 利用多 kernel 联合 autotune 优化 triton MoE 实现。
- 通过为不同类型的 KV cache 设置不同 page size 优化 DeepSeek-V3.2 及类似模型。
- 优化前缀缓存的查询时间。
- 增加 CHITULOGSTACK_TRACE 调试用环境变量。
- 向请求返回前缀缓存命中统计信息。
- 内建 profiling 功能支持 PD 分离时 profile P (prefill) instance。
- 清除实时监控中的部分无效信息。
- 修复若干 prefill chunk size 较大时的 int32 越界问题。
- 修复 DeepSeek-V3.2 等模型中 indexer 模块的软 FP8 实现。
- 修复 PD 分离时 Prefill 结点采用 DP 并行时可能导致的死锁问题。
- 修复 scheduler 中状态不一致可能导致的死锁问题。
- 清除 FlashMLA 集成中引入的不必要 padding。
- 修复 Qwen3.5/3.6 开启 PP 时的一处问题。
- 修复 warmup 期间 DeepSeek-V3.2 等模型中 indexer KV cache 占用过大的问题。
------
- New models: Qwen3.6-27B, Qwen3.6-27B-FP8, Qwen3.6-35B-A3B, Qwen3.6-35B-A3B-FP8, DeepSeek-V4-Flash-Base (experimental only)、DeepSeek-V4-Flash-FP8 (experimental only).
- All "Qwen3_5" in model names for selection are now changed to "Qwen3.5".
- The "required" field for parameters in tool descriptions in tool-calling requests are no longer required.
- Most of the built-in CUDA kernels in Chitu now supports Hygon platforms.
- New integration for multiple new operators on Hygon platforms.
- infer.fusesharedexperts argument is now compatible with EP.
- Indexer cache in DeepSeek-V3.2 and similar models can now be cached by prefix cache.
- GLM-5/5.1-FP8 is now compatible with infer.mla_absorb=absorb-without-precomp argument.
- MTP is now compatible with infer.schedule_overlap optimization.
- (TP×DP)+(ETP×EP) hybrid parallelism.
- Fewer CPU-GPU synchronizations in DeepEP + DeepGEMM prefill steps.
- Optimization on prefix cache hit rate on request routing among DP ranks and instances.
- Optimization on triton MoE implementations via co-auto-tune on multiple kernels.
- Optimization on DeepSeek-V3.2 and similar models via setting different page sizes for different types of KV caches.
- Optimization on prefix cache query time.
- New CHITULOGSTACK_TRACE environment variable for debugging.
- Add prefix cache hit status to responses.
- The built-in profiling feature now support profiling the P (prefill) instances in PD-disaggregated deployment.
- Less redundant data in monitored metrics.
- Fixed several issues on int32 overflow on large prefill chunk sizes.
- Fixed the soft FP8 implementation on indexer module in DeepSeek-V3.2 and similar models.
- Fixed a dead-lock issue on PD-disaggregation when prefill instances use DP.
- Fixed a dead-lock issue caused by inconsistent states in scheduler.
- Removed some redundant paddings from MLA integration.
- Fixed an issue on Qwen3.5/3.6 models when using PP.
- Fixed unexpected large indexer KV cache size during warming-up in DeepSeek-V3.2 and similar models.
------
Official Docker images / 官方 docker 镜像:
- 英伟达 / NVIDIA (arch 8.0, 8.9): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-nvidiaarch80_89:v0.5.6
- 英伟达 / NVIDIA (arch 9.0): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-nvidiaarch90:v0.5.6
- 沐曦 / MetaX: qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-muxi:v0.5.6
- 昇腾 / Ascend (A2): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-ascend_a2:v0.5.6
v0.5.52026-04-23
- 支持 LLaDA2.1-mini、LLaDA2.1-flash 模型,是赤兔首次支持扩散语言模型。
- 支持 Kimi-K2.6。
- 向后兼容 Qwen3-Coder-Next-FP8、GLM4.7-FP8。
- 进一步优化 DeepSeek-V3.2 及类似模型中的稀疏 attention。
- 优化 Qwen3-Next 和 Qwen3.5 模型开启 MTP 时的性能。
- 面向前缀缓存优化 DP 路由策略。
- 优化模型加载速度。
- 工具调用兼容 PD 分离。
- 支持单独控制各模块的日志级别(文档)。
- 将 mooncake 作为 pip 安装时的可选依赖,不再需要单独安装。
- 更新 flashinfer 可选依赖。
- 修复在海光平台上的一些兼容问题。
- 修复 infer.prefixchunksize 较大时的溢出问题。
- 修复多 stream 导致的显存不能及时释放的问题。
- 修复只有 PCIe 互联的环境上的 allreduce 性能。
- 删除了 infer.usecudagraph=auto 时在昇腾平台上默认关闭 graph 的一项过时判断。
- 重构算子分发逻辑。
- 重构异步调度。
- 改进代码仓库中的若干测试。
------
- Added support for LLaDA2.1-mini and LLaDA2.1-flash models, first supported diffusion LLM models.
- Added support for Kimi-K2.6.
- Added backward support for Qwen3-Coder-Next-FP8 and GLM4.7-FP8.
- Further optimized sparse attention in DeepSeek-V3.2 and similar models.
- Optimized Qwen3-Next and Qwen3.5 models when enabling MTP.
- Optimized DP routing strategy with respect to prefix caching.
- Opitmized model loading speed.
- Made tool calling compatible with PD-disaggregation.
- Supported logging level control on specific modules (doc).
- Added mooncake as an optional dependency during pip install. It no longer needed to be installed manually.
- Upgraded flashinfer optional dependency.
- Fixed some compatible issues on Hygon platform.
- Fixed overflow issues when infer.prefixchunksize is high.
- Fixed late memory freeing caused by multi-streaming.
- Fixed allreduce performance on platforms where PCIe is the only interconnect.
- Removed an out-dated default behaviour that turns off the graph when infer.usecudagraph=auto.
- Refactored operator dispatching.
- Refactored asynchornous scheduling.
- Improved multiple tests in the repository.
------
Official Docker images / 官方 docker 镜像:
- 英伟达 / NVIDIA (arch 8.0, 8.9): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-nvidiaarch80_89:v0.5.5
- 英伟达 / NVIDIA (arch 9.0): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-nvidiaarch90:v0.5.5
- 沐曦 / MetaX: qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-muxi:v0.5.5
- 昇腾 / Ascend (A2): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-ascend_a2:v0.5.5
v0.5.42026-04-09
- 新模型:GLM-5.1 系列和 Kimi-K2.5。
- 兼容 OpenAI /response API(文档)。
- 支持在 Qwen3-Next 和 Qwen3.5 系列模型中使用 MTP。
- 支持在 DP 并行时例外地 TP 并行 embedtoken 和 lmhead 层(设置 infer.embedtokenslmheadtp_size 使用)。
- 新增优雅关闭服务的功能。
- 新增内置的 profiling 功能。
- 优化 DeepSeek-V3.2 及类似模型中的稀疏 attention。
- 弃用 infer.maxreqs 选项,改为意义更明确的 infer.maxbatchsize 和 infer.maxconcurrent_requests 选项。
- 增加 infer.mla_absorb=auto 选项,并将 auto 作为默认值。
- 重构有状态的采样器。
- 重构 KV cache,使其更能适配不同种类的模型。
- 重构代码仓库中包含的若干测试。
- 修复前缀缓存中的若干问题。
- 修复 MoE 算子分发以及 DeepGEMM 集成的若干问题。
- 修复 log 不能被完整存储到文件的问题。
- 修复带有稠密层的 MoE 模型无法使用 DP+EP+PP 并行的问题。
- 修复依据 infer.memory_utilization 选项自动分配 KV cache 时的计算错误。
- 修复监控数据中显存占用量的显示错误。
- 进一步修复 tokenizer 解码特殊字符时的问题。
------
- New models: GLM-5.1 series and Kimi-K2.5.
- Compatibility with OpenAI /response API (doc).
- MTP support in Qwen3-Next and Qwen3.5 series models.
- Support of exceptionally using TP for embedtoken and lmhead layer when using DP for other layers (set infer.embedtokenslmheadtp_size to use).
- New feature to gracefully shut down the service.
- New built-in profiling support.
- Optimize sparse attention in DeepSeek-V3.2 and similar models.
- Deprecating infer.maxreqs argument and replacing with more explicit infer.maxbatchsize and infer.maxconcurrent_requests arguments.
- New infer.mla_absorb=auto argument, where auto is the new defult value.
- Refactoration of samplers with states.
- Refactoration of KV cache to better support different types of models.
- Refactoration of some tests in the repo.
- Fixes on prefix caching.
- Fixes on MoE operator dispatching and DeepGEMM integration.
- Fix on the issue that logs cannot be fully saved to files.
- Fix on compatibility between MoE modes with dense layers, and DP+EP+PP parallelism.
- Fix on the calculation when automatically allocating KV cache with respect to infer.memory_utilization.
- Fix on device memory usage in statistics.
- Further fix on tokenizer decoding of special characters.
------
Official Docker images / 官方 docker 镜像:
- 英伟达 / NVIDIA (arch 8.0, 8.9): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-nvidiaarch80_89:v0.5.4
- 英伟达 / NVIDIA (arch 9.0): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-nvidiaarch90:v0.5.4
- 沐曦 / MetaX: qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-muxi:v0.5.4
- 昇腾 / Ascend (A2): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-ascend_a2:v0.5.4
- 昇腾 / Ascend (A3): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-ascend_a3:v0.5.4
v0.5.32026-03-27
- 支持 Qwen3.5 系列模型。
- 支持前缀缓存(设置 infer.enableprefixcaching 选项)。
- 监控数据支持 Grafana 可视化。
- /tokenize 接口新增包含 chat template 的变体。
- 改进 Anthropic 接口的兼容性。
- DeepSeek-V3.2 及类似模型中的 indexer 模块支持 soft FP8(设置 infer.raiselowerbitfloatto 选项)。
- 升级 NVIDIA 镜像的基础软件版本。
- 重构 VL 多模态模型支持。
- 重构对不同 moe 算子实现的分发逻辑。
- 确保 ue8m0 scale 的处理与 DeepSeek 模型的原始定义一致。
- 优化 DeepGEMM 接入过程中的若干开销。
- 修复量化算子除以 0 的问题。
- 修复 EP 通信算子与共享专家算子的重叠优化。
- 修复 moe 临时激活值显存均衡功能错误地将阈值设置得过低的问题。
- 修复个别算子中的 int32 溢出问题。
- 修复 tokenizer 解码特殊字符时的若干问题。
- 修复流水线并行时对共享 embedding - lm_head 层的处理。
- 避免在构建镜像时在工作目录中留下 root 权限的临时文件。
------
- New models: Qwen3.5 series.
- Prefix caching (set infer.enableprefixcaching argument).
- Visualizing monitoring metrics with Grafana.
- New variant of /tokenize endpoint that includes chat templates.
- Improved compatibility with Anthripic API.
- Soft FP8 support for indexer module in DeepSeek-V3.2 and similar modules (set infer.raiselowerbitfloatto argument).
- Upgraded base dependencies in NVIDIA image build.
- Refactoration of VL multimodal models.
- Refactoration of dispatching to different implementation of MoE operators.
- Treating ue8m0 scales in the same way as model definition in DeepSeek models.
- Reduced overhead on DeepGEMM integration.
- Fix on a dividing-by-0 issue in some quantzation operators.
- Fix on overlaping EP communication and shared expert computation.
- Fix on a too-low threashold for MoE temporary activation memory balancing.
- Fix on a int32 overflow issue in some operators.
- Fix on tokenizer decoding for sprcial characters.
- Fix shared embedding - lm_head layer for Pipeline Parallelism.
- No longer generating temporary files in root permision in working directory when building images.
------
Official Docker images / 官方 docker 镜像:
- 英伟达 / NVIDIA (arch 8.0, 8.9): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-nvidiaarch80_89:v0.5.3
- 英伟达 / NVIDIA (arch 9.0): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-nvidiaarch90:v0.5.3
- 沐曦 / MetaX: qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-muxi:v0.5.3
- 昇腾 / Ascend (A2): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-ascend_a2:v0.5.3
- 昇腾 / Ascend (A3): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-ascend_a3:v0.5.3
v0.5.22026-03-12
- 工具调用支持。
- 新模型:GLM-4.6V、GLM-4.7-Flash、GLM-5、Qwen3-Coder-Next、DeepSeek-V3.2(非 Exp)。
- 支持在 DeepSeek-V3.2 及同类模型使用 FP8 KV cache。
- 针对 OpenAI 和 Anthropic 接口兼容性的多项改进。
- 改进 Triton MoE kernel 的 auto-tuning。
- 针对显存分配器在显存压力较大时隐式性能下降的问题增加警告。
- 增加记录请求 trace 的调试选项。
- 增加昇腾 MLA 优化算子。
- 沐曦 native layout kernel 兼容性更新。
- 针对 MTP 多项修复。
- 针对 KV cache 管理的多项修复。
- 针对 DeepSeek-V3.2、Qwen3-VL 等模型的修复。
- 修复 Interrupt(Ctrl+C)有时无法停止赤兔的问题(#127)。
- 修复针对低速 I/O 场景的分别进行模型权重预处理和模型推理的功能。
- 重构 PD 分离。
- 重构流水线并行(PP)。
- 重构 Qwen3-Next 及同类模型中的权重合并优化。
- 重构镜像构建脚本。
------
- Tool calling support.
- New models: GLM-4.6V, GLM-4.7-Flash, GLM-5, Qwen3-Coder-Next, DeepSeek-V3.2 (w/o -Exp).
- Support using FP8 KV cache in DeepSeek-V3.2 and similar models.
- Multiple improvements on OpenAI and Anthropic API compatibility.
- Improvement on auto-tuning on Triton MoE kernels.
- New warnings on silent performance degration on high device memory pressure.
- New debugging option for saving request trace.
- New optimized kernel for MLA on Ascend platforms.
- Compatibility update on native layout kernels on MetaX platforms.
- Multiple fixes on MTP.
- Multiple fixes on KV cache management.
- Fix on DeepSeek-V3.2 and Qwen3-VL.
- Fix on the issue that interruption (Ctrl+C) sometimes unable to stop Chitu (#127).
- Fix on the separated model preprocessing and model inference feature for slow I/O platforms.
- Refactoration on PD-disaggregation.
- Refactoration on Pipeline Parallelism (PP).
- Refactoration on weight merging for Qwen3-Next and similar models.
- Refactoration on image-building scripts.
------
Official Docker images / 官方 docker 镜像:
- 英伟达 / NVIDIA (arch 8.0, 8.9): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-nvidiaarch80_89:v0.5.2
- 英伟达 / NVIDIA (arch 9.0): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-nvidiaarch90:v0.5.2
- 沐曦 / MetaX: qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-muxi:v0.5.2
- 昇腾 / Ascend (A2): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-ascend_a2:v0.5.2
- 昇腾 / Ascend (A3): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-ascend_a3:v0.5.2
v0.5.12026-02-06
- 初步支持摩尔线程 GPU。
- MTP(multi-token prediction)支持。
- 优化开启 PP 或 DP 时的任务预处理性能。
- 针对 TP、EP 并行以及 MoE 类模型(特别是 Qwen3-Next 和 DeepSeek-V3.2 模型)进行算子融合和优化。
- 支持 Qwen3-VL 系列模型。
- 支持 FP8 KV cache 量化。
- 降低 MoE 模型在 prefill 时的峰值显存占用。
- 降低模型加载时的主存占用。
- 改进多机任务启动脚本。
- 性能测试脚本支持更多场景。
- 改进服务启动时的 auto tuning。
- 初步兼容 Anthropic API。
- 修复关于任务调度、KV cache 管理、性能监控等的多处问题。
- 对 MoE、元信息通信、异构 attention 的 KV cache 管理、算子性能测试等进行重构。
--------------
- Initial support for MooreThreads GPUs.
- MTP(multi-token prediction)support.
- Optimize task preprocessing when PP or DP is enabled.
- Operator fusion and optimization tageting TP, EP parallelism and MoE models (especially Qwen3-Next and DeepSeek-V3.2).
- Support Qwen3-VL model series.
- Support FP8 KV cache quantization.
- Reduce peak device memory usage when prefilling MoE models.
- Reduce main memory usage when loading models.
- Improve multi-node launching scripts.
- Cover more cases in benchmarking scripts.
- Improving auto-tuning when starting service.
- Initial support for Anthropic API.
- Multiple bug fixes for task scehduling, KV cache management, performance monitoring, etc.
- Multiple refactoration for MoE, metadata communication, operator benchmarking, etc.
--------------
Official Docker images / 官方 docker 镜像:
- 英伟达 / NVIDIA (arch 8.0, 8.9): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-nvidiaarch80_89:v0.5.1
- 英伟达 / NVIDIA (arch 9.0): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-nvidiaarch90:v0.5.1
- 沐曦 / MetaX: qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-muxi:v0.5.1
- 昇腾 / Ascend (A2): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-ascend_a2:v0.5.1
- 昇腾 / Ascend (A3): qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-ascend_a3:v0.5.1
v0.5.02025-12-12
针对集群部署性能的多项改进:
- 更好的 DP+TP+EP 混合并行支持。
- MoE 负载均衡策略。
- 针对预处理和后处理的性能优化。
- 多处问题修复。
Multiple improvements on cluster deployments:
- Better support on hybird DP+TP+EP parallelism.
- Load balancing strategy for MoE.
- Optimizations on pre-processing and post-processing.
- Multiple bug fixes.
---
Official Docker images / 官方 docker 镜像:
- NVIDIA: qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-nvidia:v0.5.0
- Muxi: qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-muxi:v0.5.0
- Ascend A2: qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-ascend:v0.5.0
- Ascend A3: qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-ascend-a3:v0.5.0
v0.4.32025-09-18
Fixed some performance issues.
修复了一些性能问题。
---
Official Docker images / 官方 docker 镜像:
NVIDIA: qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-nvidia:v0.4.3
Muxi: qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-muxi:v0.4.3
Ascend: qingcheng-ai-cn-beijing.cr.volces.com/public/chitu-ascend:v0.4.3