IDC's FutureScape 2026 forecast, buried in plain sight: by 2027, G2000 enterprise AI agent use increases tenfold — and the token and API-call load those agents generate increases a thousandfold. That is not a typo. And the rack has to absorb all of it.
The math is the part that matters. Per-token cost is falling 10× per year, but token volume is growing 100× faster. The aggregate spend curve is steeply up.
GPT-4 launched at $30–36 per million tokens in March 2023. By April 2026, the market floor is $0.10 per million (Gemini 2.5 Flash-Lite). That's 300–360× deflation in 37 months — faster than PC compute or dotcom bandwidth.
IDC's same forecast: token and API-call loads grow 1,000× by 2027 vs the 2024 baseline. 10× more agents × 100× hungrier per agent = the 1,000× headline number.
10× cheaper per token × 1,000× more tokens = 100× aggregate spend growth — even after the 87% per-action efficiency gain. The cost of intelligence is falling. The budget for intelligence is exploding.
IDC: $68B+ per year to deliver tokens to agents by 2029. 3.7 TeraTokens per day. And that's at the new, deflationary per-token price.
The agent workload is fundamentally different from a single chat request. A single agent workflow might fire 30, 100, even 1,000 model calls. Each call may carry long context, tool use, and memory state. The inference rack has to be designed for that pattern, not for one-shot prompts.
| Dimension | Chat / API inference (2023) | Agent inference (2027+) |
|---|---|---|
| Calls per user action | 1 call per prompt | 30–1,000+ calls per workflow |
| Context length | 4K–32K tokens | 128K–1M+ tokens, with state |
| Latency budget | 300–2,000 ms total | Multi-step, partial budgets per step |
| State management | Stateless | Long-running, memory, tool calls |
| Throughput pattern | Bursty, human-paced | Continuous, machine-paced |
| Cost per user action | ¢ | $ — even with deflation |
Agent inference is not just "more of the same." It's a different workload pattern, and it demands a different rack profile.
Prefill (heavy compute) and decode (memory-bound) split across different accelerator pools. NVIDIA Dynamo, vLLM, SGLang all converge on this architecture for MoE + agent workloads.
MoE models with 64+ experts need all-to-all GPU communication. That's why the NVL72's 1.8 TB/s per-GPU NVLink exists — to make MoE inference viable in a single rack domain.
Agents carry state. KV cache sizes explode with context. Inference racks need massive HBM (the NVL72 ships 30 TB of fast memory per rack) and KV-cache offload to keep latency flat.
Reasoning models deliberately spend more compute per answer. Inference racks need to scale horizontally within the rack for a single user request — not just across users.
Agents live close to data, close to users, close to systems of record. Inference racks deploy in regions, on edges, on-prem — not in one centralized training cluster.
At 100× aggregate spend growth, enterprises need per-token, per-agent, per-workflow cost tracking. The inference rack ships with telemetry that didn't exist three years ago.
"The use of AI agents by G2000 companies is expected to increase tenfold, with agent-related API call loads rising a thousandfold."
— IDC FutureScape 2026, Prediction 2: Worldwide IT Industry"By 2029, the number of actively deployed AI agents will exceed 1 billion worldwide — 40 times more than in 2025. These agents will execute over 217 billion actions per day and consume 3.7 TeraTokens of calls daily."
— IDC, FutureScape 2026 follow-up"The token delivery cost worldwide for supporting all these agent actions will surpass $68 billion annually, but the cost to complete an ever more complex individual action will be 87% lower."
— IDC, on the deflation-then-explosion paradox