NVIDIA just dropped some numbers that should make anyone in the infrastructure space stop and think.
AI Agents are already a production reality — and they are far hungrier for resources than any conventional chatbot. According to OpenRouter data, agentic workloads consume up to 15 times more tokens than a simple chat conversation. This is not a minor technical detail — it is a complete paradigm shift in how we think about AI infrastructure.
And that is exactly where the Vera Rubin NVL72 comes in. NVIDIA revealed that the new system delivers up to 30x more throughput per megawatt than the previous generation, the GB300 NVL72, specifically on agentic workloads. We are not talking about an incremental improvement — this is a leap that changes the math for AI factories operating in power-constrained environments. In this article, you will understand why AI agents demand so much, what the real performance numbers mean in practice, and which technologies are behind the efficiency NVIDIA is bringing to market. 🚀
Why are AI Agents so different from a regular chatbot?
Before diving into the Vera Rubin NVL72 numbers, it is worth understanding the problem it is trying to solve. A conventional chatbot has a relatively straightforward flow: the user sends a message, the model processes it and returns a response. The context is short, the interaction is one-off, and the token volume per session is pretty well-contained. In those cases, input and output sequences typically fall between 1,000 and 8,000 tokens. But AI Agents work in a completely different way. They do not just respond — they plan, execute steps, verify results, call external tools, spawn sub-agents, and iterate on themselves until they reach an objective.
Think of a concrete example: an AI agent researching a company for an investment decision. It queries financial databases, searches for news and regulatory filings, spins up a sub-agent to run competitor comparisons and model valuations, and then synthesizes everything into a final recommendation. Agents and sub-agents keep reasoning until the task is complete, and each step accumulates tokens that become the input for the next step. That is why long context handling becomes absolutely central to agentic AI performance.
According to data shared by NVIDIA based on OpenRouter information, agentic workloads consume up to 15 times more tokens than traditional chat interactions. In practice, this means a single autonomous agent running a complex task can generate a processing volume equivalent to dozens of simultaneous conversations on a conventional system. And this same pattern repeats across virtually every agentic use case, from software development to customer service to deep research. For anyone operating AI infrastructure at scale, this number is an important wake-up call: throughput demand is going to grow much faster than the growth in end-user numbers might suggest.
Receive the best innovation content in your email.
All the news, tips, trends, and resources you're looking for, delivered to your inbox.
By subscribing to the newsletter, you agree to receive communications from Método Viral. We are committed to always protecting and respecting your privacy.
This behavior creates a serious bottleneck. If the infrastructure is not designed to handle this type of load, what you get is high latency, long request queues, and in the worst case, complete degradation of the experience. Companies already migrating their workflows to agent-based architectures are feeling this firsthand. The question is no longer whether AI Agents will dominate enterprise use cases — it is when. And the answer to that when depends directly on how fast infrastructure can evolve to support this new reality.
Vera Rubin NVL72: 30x more throughput per megawatt and 35x lower cost per token
The Vera Rubin NVL72 is NVIDIA’s latest system architecture, and it was designed with this agentic scenario in mind from the ground up. The most impactful number the company shared is a 30x improvement in throughput per megawatt compared to the previous generation, the GB300 NVL72, specifically when the workload is agentic. These numbers were measured using the SemiAnalysis AgentX workload, composed of real recorded agentic coding sessions with real context growth, tool calls, and preserved sub-agent creation. In other words: this is not a synthetic test — it is real production behavior.
This metric is far more relevant than raw throughput because it puts energy efficiency at the center of the equation — something any large-scale data center operator will immediately recognize as a critical cost and sustainability factor. For power-constrained AI factories, this translates directly into 30x more agentic work within the same energy budget.
But what is behind this leap? The Vera Rubin architecture represents a significant evolution on multiple fronts simultaneously. From a silicon perspective, the new generation incorporates advances in the manufacturing process and in the organization of the memory hierarchy, allowing models to process much longer contexts with far less latency. In agentic sessions, context accumulates across steps and can reach hundreds of thousands of input tokens, with wide variation in both input and output size between requests. That is why performance measurement had to evolve to capture the complete agent flow, not just a single inference request.
The Blackwell platform as a solid foundation
On the SemiAnalysis AgentX benchmark, the NVIDIA Blackwell platform delivers leading performance across multiple agentic models, including Kimi K3, MiniMax M3, GLM5.3, Qwen3.5, and DeepSeek V4 Pro. The GB300 NVL72, for example, already delivered up to 15x more throughput per megawatt than the Hopper architecture on the DeepSeek V4 Pro model, providing a high-performance foundation for running agentic workloads. This advance reflects the advantage of a larger-scale GPU domain combined with software co-designed to deliver far superior inference efficiency.
Vera Rubin extends this advantage even further, lifting performance across the entire Pareto curve to deliver up to 30x more throughput per megawatt than the GB300 NVL72 on the DeepSeek V4 Pro model. It is worth noting that these initial results, measured with the SemiAnalysis AgentX workload and still pending review by SemiAnalysis, do not even reflect the Vera CPU performance for tool calls. In other words, there is room for it to get even better. 💡
Cost per token and the impact on margins
Throughput per megawatt directly impacts the cost of each token produced. With cost per million tokens up to 35x lower than the GB300 NVL72, the Vera Rubin NVL72 can run agents continuously, at scale, covering the full range of customer workloads. For power-constrained AI factories, throughput per megawatt determines the AI factory revenue, while cost per million tokens determines the profit margin on that revenue.
It is also worth mentioning that NVIDIA DSX MaxLPS technologies manage power at the GPU, rack, and workload levels to provision up to 40% more GPUs within the same megawatt budget, pushing throughput per megawatt even further at AI factory scale. 🔥
Extreme co-design for agentic scale
Modern inference optimization spans a range of techniques that are especially critical for agentic AI. The Vera Rubin NVL72 enables all of them and then some through extreme co-design at every layer of the platform. Here are the key highlights:
- Disaggregated serving: separates context processing (prefill) from response generation (decode), allowing each stage to scale independently.
- Rate matching: synchronizes the speeds at which prefill and decode GPUs produce tokens, maximizing efficiency.
- Large-scale expert parallelism: distributes expert sub-networks in mixture-of-experts models across the scale GPU domain.
- Distributed KV-caching: extends memory across the GPU domain, while KV-cache offloading moves less active contexts to host and storage, keeping already-processed context accessible without recomputation.
- KV-aware routing: directs incoming requests to GPUs that already have the relevant cached context, reducing redundant computation in long sessions.
- Fused CUDA kernels: techniques like MegaMoE combine many compute and inter-GPU communication operations into a single execution step, keeping GPUs active instead of waiting for data.
Rubin GPUs feature enhanced fifth-generation Tensor Cores and the third-generation Transformer Engine, which accelerate both prefill and decode inference. Meanwhile, NVFP4 quantization compresses model weights to 4-bit precision, reducing memory usage and increasing throughput without sacrificing output quality.
The NVL72 scale domain
The NVL72 scale domain, an architecture that defines both Vera Rubin and Grace Blackwell, enables the high-bandwidth, low-latency GPU-to-GPU communication that is essential for techniques like large-scale expert parallelism and distributed KV-caching. Purpose-built to feed this domain, NVIDIA NVLink interconnect technology and NVLink Switches, now in their sixth generation, deliver 10x higher packet rate and 3x lower latency than off-the-shelf Ethernet alternatives.
Spanning optimized CUDA kernels, inference runtimes like NVIDIA TensorRT LLM, and serving frameworks like NVIDIA Dynamo, NVIDIA’s software stack is co-designed with the hardware to enable all of these optimizations. And the results presented so far reflect only current performance — the complete platform is a seven-chip architecture that includes the Vera CPU, the Groq 3 LPU, the NVLink 6 Switch, the BlueField-4 DPU, the Spectrum-6 SPX, and the ConnectX-9 SuperNIC, all purpose-built for AI factories deploying agents at scale.
Why throughput per megawatt is the metric that matters now
For a long time, the go-to benchmark for evaluating AI hardware was raw throughput — how many tokens per second the system can generate, period. That metric still matters, of course, but it only tells half the story. The other half is how much you pay to generate those tokens, both in energy costs and in hardware capital costs. That is why throughput per megawatt is emerging as the most relevant metric for anyone making infrastructure decisions in production environments with AI Agents and large-scale language models.
Let us be direct: a 30x gain on this metric is not just a footnote on a presentation slide. It changes the fundamental math of how you design, operate, and price an AI factory. Imagine a data center that currently operates under power constraints — an extremely common scenario, given that electrical capacity expansion is not keeping up with the growth in compute demand. With the Vera Rubin NVL72, the same power envelope that previously supported a certain volume of agentic workloads can now support up to 30 times more. That means more customers, more models in production, more agents running in parallel — without needing to build a new data center or negotiate more capacity from the power grid.
NVIDIA clearly understood this shift in perspective before introducing the Vera Rubin NVL72. By positioning the product around this specific metric — and specifically for agentic workloads — the company is signaling that it knows exactly where the market is headed. The next few years will be marked by an explosion in the number of autonomous agents running in production, and each one of them will demand a compute volume that no cost model based on conventional chatbots was prepared to absorb. Talking about energy efficiency in this context is not corporate greenwashing — it is serious financial and operational planning.
That is also why the comparison with the GB300 NVL72 generation is so revealing. We are not comparing generations separated by many years — and yet the leap is 30x on the metric that matters most for anyone operating at scale. These initial Vera Rubin NVL72 results demonstrate NVIDIA’s accelerated pace of innovation, and with continuous software optimizations, the performance of both Vera Rubin and the GB300 NVL72 should keep improving. The extreme co-design also extends to co-engineering with partners, and Vera Rubin is already in full production, scaling across the entire ecosystem. For anyone following the industry closely, this kind of progress in short cycles is what defines which hardware platforms will dominate AI infrastructure in the coming years. 🧠
