Zhipu AI: GLM-Driven Infra Agent Optimizes Its Own Inference System
nashnova research
Zhipu's GLM-5.3-Flash has been deployed on a cluster of over 100,000 domestic AI accelerators, boosting end-to-end throughput roughly 3.2× — much of the optimization was done not just by human engineers but by an infrastructure agent powered by GLM itself, which chief scientist Tang Jie describes as an early form of recursive self-improvement.
What makes running a large model on 100,000 domestic chips so hard?
Domestic AI accelerators have less memory capacity and lower interconnect bandwidth than Nvidia GPUs, and their software ecosystems are still maturing — some operators (the basic compute units on a chip) are not yet fully supported.
The model also needs to handle 1-million-token long contexts and multimodal requests. This means → memory pressure and data-transfer pressure hit every chip simultaneously.
In plain terms = the hardware ceiling is lower, the software toolkit is thinner, yet the workload is just as demanding — like running the same large truck on a narrower road.
How did Zhipu work around the hardware limits?
ReplaySSM trades extra computation for memory savings; combined with intra-node tensor parallelism — splitting one layer's computation across multiple cards — this cuts per-card memory pressure.
INT8/FP8/BF16 mixed-precision caching stores different data at different precisions, raising effective capacity utilization.
At the architecture level, an EPD decoupled design (encode–prefill–decode, each stage scheduled independently) gives each phase more flexible resource allocation.
End result: hardware utilization and per-token cost are now close to mainstream Nvidia GPU platforms.
An AI optimizing its own system — what did the Infra Agent actually do?
Zhipu let GLM-5.3 power an infrastructure agent with access to "dense feedback" — correctness checks, runtime profiling data, and experimental results from proposed optimizations.
The agent found a precision-accumulation bug: TF32 rounding errors compounded as sequence length grew. The fix has been merged into the open-source Flash Linear Attention project as PR #1180.
It also pinpointed an overlap gap between KV Transfer and DeepEP scheduling — an intra-node path was not releasing Python's GIL (Global Interpreter Lock, which restricts execution to one thread at a time) promptly enough, inflating transfer overhead past 30%. After the fix, overhead dropped to below 1%.
In another case, the agent restructured a decode kernel's computation, eliminating a normalization step that was redundantly executed four times. That kernel's performance improved 1.71×.
Does this count as "recursive self-improvement"?
Chief scientist Tang Jie's framing: this is an early form of Recursive Self-Improvement (RSI) — the model helps optimize the system it runs on, and improvements to that system feed back into training and deploying the next-generation model.
He also stresses that true RSI is still a long way off; what exists now is "a loop at its smallest possible scale."
This reflects a key signal: AI is no longer just the object being optimized — it is starting to participate in optimizing its own infrastructure, even if only in an assistive role for now.
How does the engineer's role change?
Human engineers still set goals, build the feedback environment, and review high-risk changes, but their role is shifting from "the person who solves every problem directly" to "the person who designs the feedback system."
This means → the engineer's core competency is migrating from "writing code to fix bugs" toward "designing environments where an agent can discover and fix problems on its own."
Zhipu's team believes every engineering task the agent completes could become training data for the next-generation model — whether this loop can keep scaling is the key test of the approach's real value.
Real-world validation: how did the anonymous launch perform?
After deployment, GLM-5.3-Flash ran under the anonymous name "Ox-Alpha" on the OpenCode and OpenRouter platforms.
Within one week it became one of the most-used models on both platforms, processing over 62 trillion tokens in six days.
In plain terms = no brand, no marketing — just an anonymous model dropped into an open platform for blind testing, and usage spoke for itself. That is the most direct proof the deployment works.
市场有风险，内容仅供研究参考，不构成投资建议。