Many conversations around AI infrastructure and data centers still focus on GPUs, power density, and cooling. These are important considerations and deserve proper attention, but they don’t fully capture what’s driving demand.
At recent industry events, particularly NVIDIA’s GTC conference, there has been a noticeable change in how AI infrastructure is discussed. NVIDIA CEO Jensen Huang emphasized that AI is entering a new phase, in which organizations are moving from experimentation to deploying AI models within real-world workflows and applications. In practice, that shift is already showing up in how AI systems are evaluated. Model performance still matters, but so do the realities of running these systems in production, where power availability, cooling capacity, and sustained infrastructure load become defining constraints.
The Rise of the Token Economy
At a more fundamental level, AI workloads are measured in tokens, those units of data processed each time a model generates a response, completes a task, or executes a workflow. Tokens are often viewed as a technical concept, but they are increasingly used as a practical measure of AI output. They offer a better way to understand how AI systems draw on infrastructure and the financial impact that follows.
Every interaction with an AI system generates tokens. At enterprise scale, those interactions compound quickly, with more complex use cases such as analysis, automation, and agent-based workflows producing significantly higher token volumes.
As a result, new benchmarks are starting to take hold. Metrics such as tokens per second, cost per million tokens, and tokens per watt are being used to measure how AI systems operate. They reflect sustained output over time rather than peak performance. At GTC, there was a clear focus on cost per token as a primary metric for gauging AI systems in production.
From Training Models to Operating Them
For much of the past decade, AI infrastructure has been defined by training. The industry narrative has focused on building larger and more capable models using powerful GPU clusters.
As highlighted by NVIDIA’s CEO, that approach is now giving way to inference or the real-time use of AI models in production environments. Where training is episodic, inference introduces continuous demand, generating tokens each time a model is queried, integrated into an application, or used to automate a workflow.
Inference scales differently. It places sustained demand on infrastructure rather than periodic bursts tied to model training. As models are deployed into workflows, token volumes greatly increase, along with the demands on the data center infrastructure that supports them. This is how inference behaves in production mode.
What This Means for Infrastructure
With this rise in demand, infrastructure is now being evaluated on how well systems convert electricity into usable AI output, within the limitations of available power, often measured in tokens per watt. Over time, this type of metric is likely to matter as much as overall system throughput.
System design is also evolving. Infrastructure is becoming more heterogeneous, with different types of processors supporting different parts of the workload. This too was evident at GTC, where NVIDIA highlighted more integrated architectures, including the incorporation of Groq’s inference-optimized chips alongside GPUs to improve throughput and cost per token.
Taken together, these changes point to a significant transition. AI infrastructure is evolving into a platform for generating and delivering tokens in high-volume production environments.
The Economics of AI Infrastructure
As with most discussions on infrastructure, the conversation soon turns to the economics of implementing and operating these systems.
Improvements in hardware and software are lowering the cost to generate each token, as inference workloads produce tokens continuously. This combination brings cost into sharper focus. In such a model, tokens effectively become the unit of cost, linking infrastructure performance directly to financial outcomes.
Recent analysis from Gartner reinforces this dynamic, noting that while the cost of inference is expected to decline significantly over the next several years, overall demand is likely to increase as AI systems become more capable and widely deployed. More advanced use cases require more tokens per interaction, offsetting any efficiency gains.
This trend is surfacing in how organizations pay for AI. Cloud-based platforms, particularly those accessed through API-driven, usage-based pricing, provide flexibility and speed to deployment. They also introduce variable pricing tied directly to token consumption. For companies running sustained inference workloads, those costs can escalate quickly as token volumes grow. In many instances, continuing to “rent” tokens will become one of the more expensive ways to operate AI over time.
Many organizations are already evaluating alternative approaches. On-premise and colocation-based AI infrastructure offer more predictable cost structures for workloads that run continuously. These environments require upfront investment but provide greater long-term control over operating costs.
This pattern is familiar. Early cloud adoption was driven by flexibility and speed. Over time, organizations moved toward more balanced approaches that combine cloud and owned infrastructure. AI is likely to follow a similar path. Some workloads will remain in the cloud, while others will move closer to environments where cost and performance can be managed more directly.
Reframing AI Capability and Costs
AI infrastructure and data centers are often discussed in terms of key components such as GPUs, power, and cooling. In reality, what matters more is the volume of tokens being generated and the underlying foundation required to support them.
This changes how data center readiness is assessed.
It is less about peak performance and more about sustained operation. Systems need to deliver AI output reliably and efficiently over time.
The question then is no longer what AI systems are capable of, but what it costs to operate them in the long run. Organizations that understand that distinction early will be better positioned to make the right data center and infrastructure decisions as AI becomes further embedded in day-to-day operations.