Skip to main content

Inside Inspur's 40,000-Agent Single-Rack Deployment: Native CPU Liquid Cooling and Multi-Model Teaming

On July 13, 2026, Inspur Information used the 2026 Open Compute China (OCC) conference to drop three coordinated products: the industry's first CPU-native liquid-cooled rack-scale server, a multi-model fusion API, and the YuanNao SD200 supernode enterprise edition. The unifying message — AI infrastructure's core mission has shifted from "serving large-model inference" to "running massive Agent fleets" and "producing high-quality Tokens continuously." Here is a full technical breakdown.

1. Why Agent-Era Infrastructure Must Be Rebuilt​

IDC puts the Chinese enterprise AI Agent market at roughly RMB 19 billion in 2025, with a CAGR above 110% from 2025 to 2028. Gartner goes further: by 2026, 40% of enterprise applications will integrate task-oriented AI Agents.

But Agent workloads are fundamentally different from inference:

  • Inference: one input, one output, finished;
  • Agents: task decomposition, tool use, multi-turn collaboration, long-running state — possibly an entire swarm of Sub-Agents running in parallel.

Inspur VP Zhao Shuai noted that compute ratios in the AI industry are shifting from "GPU-centric" to "heterogeneous, collaborative." Agent workloads involve heavy integer arithmetic, control flow, state machines, and orchestration — all of which naturally land on CPUs.

Power density is also exploding: domestic AI racks are pushing 300 kW this year, with global racks already entering the megawatt class. Traditional air cooling caps at 40–50 kW per rack — liquid cooling is no longer optional.

2. Product One: CPU-Native Liquid-Cooled Rack-Scale Server​

2.1 Headline Specs​

  • CPUs per rack: up to 384 (open OCM architecture, x86 + ARM compatible);
  • Agents per rack: 40,000+ simultaneously;
  • Scale-up vs. April release: 40× the density of the "Qi Qianxia" solution (which ran 1,000 OpenClaw Agents on a single 2U server).

2.2 Why OCM Matters​

The Open Compute Module (OCM) architecture treats the whole rack as a container for compute modules:

  • Mix-and-match different CPU generations and ISAs (x86, ARM);
  • No need to redesign the rack for every new chip generation;
  • Rack R&D cycle compressed, enterprises can upgrade per-CPU as needed.

2.3 Rack-Scale "Native Liquid Cooling"​

Traditional liquid cooling is a hybrid: cold plates on CPUs only, with fans handling memory, NICs, and SSDs. Inspur flipped the playbook:

  • Full cold-plate coverage: memory, NICs, optical modules, and SSDs are all part of the liquid loop;
  • 2U ultra-thin compute sleds: 16 CPUs per node, with all I/O laid flat on the motherboard;
  • Cable-free rack: space once occupied by fans, cold pipes, and cables is fully reclaimed for compute and I/O;
  • Hot maintenance supported: zero business interruption; rack-wide O&M efficiency improved by more than 100%.

3. Product Two: Multi-Model Fusion API — Let Multiple LLMs Team Up​

Just running Agents is not enough — they also have to get smarter. Real tasks are too complex for a single model, and no amount of parameter scaling fixes specialization gaps (one model reasons well, another writes well).

Inspur's YuanNao QiZhi EPAI platform introduces a Multi-Model Fusion API, with a four-step workflow:

  1. Dispatch: send the same task to multiple candidate models in parallel, each producing its own answer;
  2. Review & fuse: a "reviewer-fuser" model compares consensus, disagreement, omissions, and unique perspectives;
  3. Unified output: synthesize one final, well-rounded response;
  4. Tiered routing: simple Q&A, tool calls, format conversion go to a lightweight single model; only complex long-chain tasks trigger multi-model orchestration.

The mechanism scores 53.9% on the DRACO benchmark — higher than any single model in the candidate pool. The API supports two integration modes: direct call like any normal model service, or wired into existing agent and development frameworks, reusing their conversation, reasoning, and tool-call pipelines.

4. Product Three: YuanNao SD200 Supernode Enterprise Edition​

Running multiple trillion-parameter models in parallel demands serious underlying horsepower. That is exactly what the SD200 supernode is for.

4.1 Key Performance Numbers​

Metric2025 Edition2026 Edition
Concurrent trillion-parameter models44
Token generation time8.9 ms4.77 ms (first to break 5 ms)
First-token latencyBaselineDown 35%

4.2 Three Software-Hardware Co-Optimizations​

  • Multi-Token Prediction (MTP): the decoder produces and verifies several candidate tokens at once, reducing the number of generation passes;
  • W4A8 precision: MoE module compute drops from BF16 to INT8, cutting memory bandwidth pressure;
  • JIT compilation: at runtime, generates dedicated GPU kernels tailored to actual tensor shapes, aligning compute with hardware characteristics.

4.3 Enterprise Edition: Lowering the Bar​

The SD200 enterprise edition trims the Scale-Up domain from 64 GPUs to 16 GPUs while still reducing trillion-model first-token latency by more than 40%. It opens the door for mid-sized enterprises that previously could only deploy hundred-billion-parameter models as auxiliary tools — they can now actually run trillion-parameter models in production.

4.4 Ecosystem Coverage​

SD200 has completed adaptation for Kimi K2.6, DeepSeek V4, GLM 5.2, MiniMax M3, and other mainstream open-source models — covering the bulk of the leading Chinese LLMs.

5. The New Division of Labor in Agent-Era Infrastructure​

This launch sends a clear message: CPU, GPU, and software platforms must work together more tightly than ever — none of the three is dispensable.

  • Software platform: model integration, task orchestration, resource scheduling, permission governance, result fusion;
  • CPU: hosting Agent instances, tool calls, sandbox execution, business system interaction;
  • GPU: handling model inference and Token generation.

Drop any link in this chain and the entire Agent application stalls. The competitive focus has therefore shifted from "who supports a single model best" to "who delivers the best system-level coordination."

6. Three Takeaways for Enterprises Deploying Agents​

  1. Recompute the compute ratio: GPU is no longer the only center. CPU compute, memory, and I/O must be re-balanced for Agent workloads.
  2. Multi-model coordination beats single-model scaling: once tasks cross a complexity threshold, fusing outputs from multiple specialized models is more stable and more cost-effective than brute-forcing one super-large model.
  3. Liquid cooling is no longer optional: with single-rack density breaching 50 kW and racing toward 300 kW — even megawatt class — native liquid cooling has moved from "energy-saving bonus" to "entry ticket for the infrastructure race."

7. Closing Thoughts​

Inspur did not put "a stronger LLM" on the marquee — it put the two engineering problems every real Agent deployment actually faces: scalable, stable runtime, and multi-model collaborative intelligence. The OCM architecture, native liquid cooling, multi-model fusion, and the SD200 supernode enterprise edition together form an "operating-system-grade" foundation for the Agent era.

The moat of any Agent application has never been the model alone — it has always been the layer underneath it that keeps the model running.

References:

  1. QbitAI: Inspur Information runs 40,000 Agents in one rack and lets large models answer questions as a team (qbitai.com, 2026-07-13)
  2. Inspur Information OCC 2026 product launch
  3. Inspur Information official: YuanNao QiZhi EPAI platform technical whitepaper
  4. Inspur Information official: YuanNao SD200 supernode product documentation