Novita AI has unveiled a suite of high-impact optimizations for the GLM4-MoE model, demonstrating how strategic architectural tweaks can drastically improve real-world AI performance. By integrating these improvements into SGLang—an advanced framework for serving large models—the team achieved a 65% reduction in Time to First Token (TTFT). This metric is crucial because it measures how quickly an AI begins responding to a user's prompt, directly impacting the perceived "snappiness" of applications. One standout technique, Shared Experts Fusion, optimizes the Mixture-of-Experts (MoE) architecture. In this setup, a model routes information through specialized "expert" pathways rather than using its entire brain for every task. By merging "shared" experts with "routed" ones, the system utilizes hardware more efficiently, particularly on high-end NVIDIA H200 chips. Additionally, the team introduced Async Transfer, which moves data between nodes in the background while the processor is busy, preventing the "stuttering" often seen in complex, multi-layered models during heavy workloads. The research also addresses the specific needs of AI agents used for programming tasks. By using Suffix Decoding—a method that guesses future words based on repeating patterns found in previously generated code—the system produced text 22% faster without needing a separate draft model. These battle-tested strategies provide a vital blueprint for developers looking to balance massive throughput with the low latency required for interactive AI tools.

Novita AI has unveiled a suite of high-impact optimizations for the GLM4-MoE model, demonstrating how strategic architectural tweaks can drastically improve real-world AI performance. By integrating these improvements into SGLang—an advanced framework for serving large models—the team achieved a 65% reduction in Time to First Token (TTFT). This metric is crucial because it measures how quickly an AI begins responding to a user's prompt, directly impacting the perceived "snappiness" of applications. One standout technique, Shared Experts Fusion, optimizes the Mixture-of-Experts (MoE) architecture. In this setup, a model routes information through specialized "expert" pathways rather than using its entire brain for every task. By merging "shared" experts with "routed" ones, the system utilizes hardware more efficiently, particularly on high-end NVIDIA H200 chips. Additionally, the team introduced Async Transfer, which moves data between nodes in the background while the processor is busy, preventing the "stuttering" often seen in complex, multi-layered models during heavy workloads. The research also addresses the specific needs of AI agents used for programming tasks. By using Suffix Decoding—a method that guesses future words based on repeating patterns found in previously generated code—the system produced text 22% faster without needing a separate draft model. These battle-tested strategies provide a vital blueprint for developers looking to balance massive throughput with the low latency required for interactive AI tools.