Cloudflare transforms its AI platform into an inference layer built specifically for agents
Cloudflare just changed the game for anyone building with artificial intelligence. And this time, the shift isn’t incremental — it’s structural.
The AI model landscape is constantly evolving, and what works today as the best model for a given task could be completely different three months from now, coming from an entirely new provider. New releases drop every week, benchmarks shuffle positions, and providers that didn’t even exist six months ago are already showing up as top picks in specific categories. Anyone building real applications with AI knows exactly what it feels like to rewrite integrations because the model that offered the best bang for the buck simply stopped being the right choice.
This isn’t an exaggeration — it’s the reality for anyone on the front lines building AI applications right now. And when it comes to AI agents, the complexity ramps up significantly. Unlike a simple chatbot that makes a single inference call per user prompt, an agent can chain ten calls in a row to complete a single task, involving different models and different providers at the same time. Your customer support agent, for example, might use a fast, cheap model to classify the user’s message, a large reasoning model to plan its actions, and a lightweight model to execute individual tasks. Latency, cascading failures, distributed costs, and a lack of centralized visibility are real problems that stall development and turn what should be a competitive advantage into an operational nightmare.
This is exactly the scenario that Cloudflare’s AI Gateway was built to solve once and for all 🎯
Cloudflare officially announced its transformation into a unified inference layer designed specifically for agents, bringing together over 70 models from more than 12 providers under a single API, with centralized cost controls, automatic failover, support for custom models, and latency optimized through its global network of data centers across 330 cities. Let’s break down what actually changes for anyone building with AI today 👇
The real problem facing anyone building agents with AI
Before diving into what the AI Gateway offers, it’s worth understanding why this solution makes so much sense right now. When you’re working with a single model and a single provider, complexity is still manageable. You set up your API key, define your prompts, test the behavior, and move on. But as soon as your project starts to grow — or as soon as you decide to use the best available model for each stage of your pipeline — everything changes. Suddenly, you’ve got API keys from five different providers, retry logic scattered throughout your code, logs in different places, and zero consolidated visibility into what’s happening with your inference calls.
With agents, this problem multiplies in a scary way. A modern agent isn’t a straight line — it’s a decision graph. It might call one model to reason about a task, another to generate code, another to validate the result, and yet another to format the final response. Each of those calls has its own latency, its own cost, and its own failure point. A simple chatbot that takes a prompt and returns a response deals with a single inference call. An agent that chains ten calls in a row turns that 50ms delay from a slow provider into 500ms of accumulated lag. A request that fails isn’t just a retry — it’s a cascade of failures that can bring down the entire execution chain.
According to data cited by Cloudflare, companies today use an average of 3.5 models from multiple providers, which means no single provider can offer a complete picture of AI usage and spending. Without a centralized layer, answering questions like how much each task completed by an agent is costing or which step is the slowest requires significant manual work — and many teams simply don’t have that kind of time.
Receive the best innovation content in your email.
All the news, tips, trends, and resources you're looking for, delivered to your inbox.
By subscribing to the newsletter, you agree to receive communications from Método Viral. We are committed to always protecting and respecting your privacy.
A unified catalog and a single endpoint for everything
The big news from Cloudflare is that developers can now call third-party models using the same AI.run() binding they already use for Workers AI. In practice, this means switching from a Cloudflare-hosted model to one from OpenAI, Anthropic, or any other provider is a one-line code change. No need to reconfigure integrations, swap out libraries, or rewrite call logic.
For those not using Workers, Cloudflare also announced that REST API support will be rolling out in the coming weeks, giving access to the full model catalog from any development environment.
The catalog already includes more than 70 models from over 12 providers, all accessible through a single API, a one-line switch between them, and a single set of credits for payment. And the provider lineup is impressive: Alibaba Cloud, AssemblyAI, Bytedance, Google, InWorld, MiniMax, OpenAI, Pixverse, Recraft, Runway, and Vidu are among those making their models available through the AI Gateway. An important detail is that the offering goes beyond text models, now including image, video, and speech models, opening the door for truly multimodal applications.
Accessing all your models through a single API also means you can manage all your AI spending in one place. With the ability to include custom metadata in requests, you can get cost breakdowns across the attributes that matter most to your business — like spending by free versus paid users, by individual customers, or by specific workflows within your application.
Bring your own model
The AI Gateway gives you access to models from all partner providers, but sometimes you need to run a model that’s been fine-tuned on your own data or optimized for your specific use case. For that, Cloudflare is working on a feature that lets users bring their own models to Workers AI.
The company revealed that the vast majority of its traffic already comes from dedicated instances for Enterprise customers running custom models on the platform, and the goal now is to democratize that access. To make this possible, Cloudflare is using Cog technology from Replicate, which simplifies the containerization of machine learning models.
Cog was designed to be straightforward: just define your dependencies in a config file and your inference code in a Python file. The tool abstracts away all the complexity of packaging ML models — things like CUDA dependencies, Python versions, and weight loading. After building the container image, you push it to Workers AI, and Cloudflare handles the deployment and delivery of the model, which becomes accessible through the standard Workers AI APIs.
The team is also working on client-facing APIs and wrangler commands to make pushing containers easier, along with faster cold starts through GPU snapshotting. This feature is currently being tested internally with Cloudflare teams and select external customers who are helping shape the product direction.
The fastest path to the first token
If you’re building live agents where the user is waiting for a response in real time, the perception of speed depends heavily on time to first token — meaning how quickly the agent starts responding, not necessarily how long the full response takes. Even if the total inference takes 3 seconds, receiving the first token 50ms faster makes the difference between an agent that feels snappy and one that feels stuck.
Cloudflare’s network, with data centers in 330 cities around the world, means the AI Gateway is positioned close to both users and inference endpoints, minimizing network time before streaming begins. When you use Cloudflare-hosted models on Workers AI, like Kimi K2.5 and real-time voice models, there’s no extra hop over the public internet — code and inference run on the same global network. This delivers the lowest possible latency for your agents.
Resilience with automatic failover
When it comes to building agents, speed isn’t the only factor that matters. Reliability is equally critical. Every step in an agent’s workflow depends on the steps before it. Reliable inference is crucial because a single failed call can compromise the entire execution chain.
Through the AI Gateway, if you’re calling a model that’s available across multiple providers and one of them goes down, Cloudflare automatically reroutes to another available provider — without you needing to write any failover logic in your code. This is especially relevant for production environments where any downtime has a direct impact on user experience.
For those building long-running agents with the Agents SDK, streaming inference calls are also resilient to disconnections. The AI Gateway buffers streaming responses as they’re generated, regardless of your agent’s lifecycle. If the agent gets interrupted mid-inference, it can reconnect to the AI Gateway and recover the response without making a new inference call or paying twice for the same output tokens. Combined with the Agents SDK’s native checkpointing system, the end user doesn’t even notice any interruption happened.
The Replicate integration
The Replicate team has officially joined Cloudflare’s AI platform team, and as the company was quick to point out, the teams don’t even consider themselves separate anymore. Integration work between Replicate and Cloudflare is in full swing, including bringing all of Replicate’s models to the AI Gateway and migrating hosted models to Cloudflare’s infrastructure. Soon, it will be possible to access models that were already popular on Replicate through the AI Gateway and host models that were deployed on Replicate directly on Workers AI.
Observability and control that make a real difference day to day
One of the biggest pain points for anyone working with inference at scale is the lack of visibility. Cloudflare’s AI Gateway provides centralized logs for all calls regardless of provider, with latency data, tokens consumed, estimated cost, and the status of each request. For anyone optimizing an agent pipeline, this is pure gold. You can pinpoint exactly which step is costing the most, which is the slowest, and where failures are happening — all in a single dashboard without needing to consolidate data from multiple provider dashboards.
The ability to set rate limits and usage policies per project is also something that makes a real difference in the real world. When you have multiple teams using the same AI infrastructure, or when you’re building a product that serves multiple customers, having granular control over who can use what — and how much — is essential to avoid billing surprises and ensure that abnormal usage in one part of the system doesn’t bring down the whole platform.
The more granular logging controls and automatic retries on upstream failures, launched in recent months alongside the dashboard overhaul and zero-config default gateways, show that Cloudflare is evolving the AI Gateway quickly and consistently, listening directly to feedback from developers already using the platform in production.
Why this matters for the future of AI agents
Cloudflare’s move with the AI Gateway is part of a larger trend shaping how AI applications will be built going forward. Provider and model fragmentation isn’t going to shrink — it’s going to grow. Every month brings new specialized models, regional providers with competitive pricing, and open source options that rival commercial APIs. Having an abstraction layer that insulates your application from this fragmentation isn’t a luxury — it’s an architectural necessity for any team that wants to stay agile and avoid being locked into a single vendor.
For those building more complex agents — like multi-agent systems where different agents communicate with each other and with different models — this unified layer is even more critical. The complexity of managing multiple integrations directly grows exponentially with the number of agents and models involved. With the AI Gateway as the central control point, that complexity is absorbed by the infrastructure, and the development team can focus on what truly matters: the business logic and agent behavior itself, not the API plumbing.
Cloudflare is clearly positioning the AI Gateway as foundational infrastructure for the next generation of intelligent applications. And given the company’s track record of building reliable, globally distributed network infrastructure, it makes perfect sense that they’re expanding that role into the world of AI inference. The market for AI development tools is still consolidating, but solutions that solve real production problems — like observability, cost control, and resilience — tend to become industry standards pretty fast. 🚀
The AI Gateway is already available for developers using the Cloudflare platform, with support for over 70 models from providers like OpenAI, Anthropic, Google, Alibaba Cloud, AssemblyAI, Bytedance, MiniMax, and others, all accessible through a single unified API. The full model catalog can be found in Cloudflare’s official documentation.
