# Workers AI and AI Gateway: running and controlling inference

> What Workers AI and AI Gateway each do, how they are billed, when hosted open models beat a model provider, and the limits to know before you rely on them.

- Published: 2026-08-28; last updated 2026-11-14
- Author: the WeStack AI team
- Subject: Cloud and DevOps
- Address: https://westack.ai/blog/cloudflare-workers-ai-and-ai-gateway

> Workers AI and AI Gateway do two different jobs: Workers AI runs open models on Cloudflare's GPUs and bills by usage, which suits embeddings, classification, speech and small to mid-sized language models that sit next to your code. AI Gateway is a proxy in front of any provider, Workers AI included, that adds logs, caching, retries, fallbacks, guardrails and spend limits. Put every model call through the gateway, and use Workers AI wherever an open model is good enough.

## Two products, two jobs

The names sit next to each other in the dashboard and get confused constantly. One is a place where inference happens. The other is a control point that inference passes through.

| | Workers AI | AI Gateway |
| --- | --- | --- |
| What it is | Open models served on Cloudflare's GPUs | A proxy between your app and model providers |
| Runs models | Yes | No |
| Works with | Its own catalogue | Workers AI, OpenAI, Anthropic, Google, Bedrock, Azure OpenAI, Mistral, Groq and others |
| You pay for | Compute used per request | Core features free; logs, guardrails and billing extras cost |
| Main risk | Model quality and rate limits for your task | Another hop, and features that are narrower than they sound |

You can use either without the other. In practice we route every call through a gateway from the first day, including the calls to Workers AI, because retrofitting observability after a cost surprise is the expensive way round.

## What does Workers AI run?

The catalogue is open models, grouped by task. At the time of writing that means language models from families such as Llama, Gemma, Qwen, DeepSeek and Mistral, embedding models such as the BGE family, a reranker, speech recognition including Whisper, text-to-speech, translation, image generation including FLUX, and small classifiers. The line-up changes often, and models are deprecated and removed, so treat any list (this one included) as a snapshot.

You call a model from a Worker through a binding (`env.AI.run()`), or from anywhere over a REST API that includes OpenAI-compatible endpoints. Around the models there is a useful set of features: function calling, JSON mode, an asynchronous batch API for jobs that can wait, a document-to-Markdown converter, and prefix caching on selected models, where repeated prompt prefixes are processed faster and billed at a discount.

Fine-tuning is limited to LoRA adapters. You train an adapter elsewhere and upload it to run against one of a handful of base models. The docs describe this as an open beta with constraints on adapter size and rank. You cannot bring an arbitrary model of your own without talking to Cloudflare.

### How it is billed

Workers AI charges for [compute used, measured in Neurons](https://developers.cloudflare.com/workers-ai/platform/pricing/). The pricing page translates that into familiar units for each model: a rate per million input tokens and per million output tokens for language models, and per image or per audio minute for others. There is a free daily allocation, then a rate per thousand Neurons on the paid Workers plan. Nothing is charged for idle time, which is the entire appeal compared with renting a GPU.

On data handling, the [data usage page](https://developers.cloudflare.com/workers-ai/platform/data-usage/) commits to not training models on customer content and, with stated exceptions, to not retaining inputs or outputs for standard inference requests. Storage features you switch on (logs, caches, vector indexes) keep what you tell them to keep.

## When does Workers AI beat calling a provider?

It wins in four situations.

**Supporting models at volume.** Embeddings, reranking, classification, moderation, transcription and translation are the plumbing of an AI product. They run many times per user action, quality differences between good open models and proprietary ones are small, and latency matters. Running them next to the Worker, with no API key and no extra network hop to another company, is simply easier.

**Spiky or low traffic.** A GPU costs the same busy or idle. If your open-model workload is ten requests a minute with occasional bursts, per-request billing beats any reserved hardware. The arithmetic for when that flips is in [FinOps for AI](https://westack.ai/blog/finops-for-ai).

**Data that should not go to a model vendor.** Some teams can accept a cloud platform's retention commitments more readily than a second vendor's.

**A small language model is enough.** Routing, extraction into a fixed schema, short summaries and intent detection rarely need a frontier model.

It loses in four as well.

**The hardest reasoning and tool use.** For multi-step agents and long, messy documents, the leading proprietary models are usually still ahead. Test on your own cases before assuming an open model will do.

**Throughput guarantees.** [Rate limits](https://developers.cloudflare.com/workers-ai/platform/limits/) are set per task type: 300 requests a minute for most text generation and 3,000 for most embedding models at the time of writing. A set of newer large models that require the paid plan get 20 requests a minute per model on standard billing and 50 on prepaid credits. That is a development limit, not a launch limit, so ask for an increase early. Requests can also wait in a capacity queue. There is an option to reject instead of waiting, which you will want for anything interactive.

**Customisation.** If your plan depends on a fully fine-tuned model or a model that is not in the catalogue, you need your own GPU capacity.

**Residency rules.** "The edge" is not a region. If you have a rule about where inference runs, get the answer in writing before you build.

A realistic split: a logistics company receives 50,000 emails a day. A small model on Workers AI classifies each one and extracts the consignment number, and an embedding model indexes it for search. Only the 6,000 that need a written reply go to a frontier model through the gateway. Most of the volume never touches the expensive model.

## What does AI Gateway add?

### Logs and cost analytics

Each request can be logged with its prompt, response, provider, status, token counts, duration and an estimated cost. You can attach up to five metadata values per request, which is how you get cost per customer or per feature. Two cautions. The cost figure is an estimate from token counts, so reconcile it against provider invoices. And logs hold prompts, which means personal data: there are per-request headers to skip logging or to store the log without its payload.

### Caching

This is the feature most often misunderstood. [The cache is exact-match](https://developers.cloudflare.com/ai-gateway/features/caching/). The key is a hash of the provider, endpoint, model, authentication header and the entire request body, so one changed character is a miss. It covers text and image responses, with a time to live of up to a month. That is valuable for fixed prompts over repeated inputs, evaluation runs and development. It does almost nothing for open-ended chat. The other techniques that cut spend are in [cutting LLM costs](https://westack.ai/blog/llm-cost-optimisation).

### Rate limits, retries and fallbacks

Rate limiting counts requests in a fixed or sliding window and returns a 429 beyond it. Retries allow up to five attempts with constant, linear or exponential backoff. Timeouts are measured to the first part of the response, so a streaming answer that has started is not cut off.

Routing is where the gateway earns its place. A dynamic route is a small versioned flow that you call in place of a model name. It can branch on metadata (paid plan or free), split traffic by percentage for an A/B test, enforce a rate or budget limit per key, and fall back to another model when one fails. You change the route in the dashboard and roll back without a deploy. There is also an automatic router that picks a cheaper model per request. We would test that against our own evaluation set before trusting it with anything customers see.

### Guardrails and DLP

Guardrails run a safety classifier (Llama Guard on Workers AI) over prompts and responses and either flag or block by category, with prompt injection as a separate category. Read the [usage considerations](https://developers.cloudflare.com/ai-gateway/features/guardrails/usage-considerations/) before you switch it on. Cloudflare puts the added latency at roughly half a second per request. Streaming is not supported: on the gateway endpoints the response is buffered, checked and returned in one piece. The classifier covers a short list of languages. Data loss prevention is separate and scans prompts and responses for patterns such as financial and identity numbers.

### Keys, billing and spend limits

You can store provider keys in Cloudflare so they are not in your application, which makes rotation a dashboard task. Or you can use Unified Billing: buy credits, let Cloudflare hold the provider credentials, and receive one bill. The [pricing page](https://developers.cloudflare.com/ai-gateway/reference/pricing/) states a 5% fee on credit purchases with provider rates passed through, and those requests have their own rate limit per gateway.

Spend limits block requests with a 429 once a budget for a model, a provider or a metadata value is used up. They are eventually consistent, so a burst of concurrent requests can overshoot. Use them as a seat belt, and keep proper alerts as well.

## How to use them together

The pattern is one gateway per application and every call through it. From a Worker the same binding reaches both kinds of model:

```ts
// A small open model on Workers AI, logged by the "support" gateway
const label = await env.AI.run(
  "@cf/meta/llama-3.1-8b-instruct-fp8",
  { messages: [{ role: "user", content: `Classify this email: ${text}` }] },
  { gateway: { id: "support" } },
);

// A provider model through the same gateway
const draft = await env.AI.run(
  "openai/gpt-4.1-mini",
  { messages },
  { gateway: { id: "support" } },
);
```

Provider models called through the binding use Unified Billing or a stored key. If you would rather keep your existing provider SDK, point its base URL at the gateway and add the gateway's authentication header.

Three habits make this worth having. Tag every request with the feature and the customer. Put the model choice in a route, not in code, so a fallback or a price change is a configuration edit. And give long agent sessions a session affinity header on Workers AI so prefix caching can hit. If you are building [agents on Cloudflare](https://westack.ai/blog/cloudflare-for-ai-agents) or a [retrieval stack](https://westack.ai/blog/rag-on-cloudflare), this is the layer both of them call.

## What are the limitations?

Check these before you commit.

1. **Exact-match cache only.** Savings depend on how repetitive your requests really are. Measure the hit rate in the first week.
2. **Guardrails change your latency and your streaming.** Budget for the delay, and decide per route whether a buffered response is acceptable.
3. **Costs are estimates.** Good for trends and allocation, not for accounting.
4. **Spend limits can be overshot** by concurrent requests.
5. **It is one more dependency.** Every model call now crosses the gateway. Know what your application does if it is unreachable, and keep a direct path you have tested.
6. **Logs are a data store.** Decide retention, access and which routes skip payload logging before the first production prompt is recorded.
7. **Workers AI limits are per account.** One noisy batch job can starve the interactive path. Use the batch API for bulk work, as covered in [scaling an LLM application](https://westack.ai/blog/scaling-llm-applications).
8. **The catalogue moves.** Pin model IDs, subscribe to the changelog, and keep an evaluation set so a forced model change is a morning's work.

## How to start this week

1. Create one gateway and send your existing provider calls through it. Change nothing else. Look at a week of logs.
2. Add metadata for feature and customer. Find your three most expensive request types.
3. For each, ask whether an open model could do it. Run fifty real examples through a Workers AI model and compare the outputs side by side.
4. Move the ones that pass. Put a fallback route behind the ones that stay.
5. Set a spend limit and an alert, then turn on guardrails only where the latency is acceptable.

The first decision is not which model to use. It is to make every model call visible, because every later decision depends on that data. This sits within the [cloud](https://westack.ai/services/cloud) and [LLMOps](https://westack.ai/services/mlops) work we do alongside [generative AI builds](https://westack.ai/services/generative-ai).

## Frequently asked questions

### What is the difference between Workers AI and AI Gateway?

Workers AI runs models: a catalogue of open models on Cloudflare's GPUs, billed by usage. AI Gateway runs nothing. It is a proxy in front of Workers AI and outside providers such as OpenAI, Anthropic and Google that adds logging, caching, rate limits, retries, fallbacks, guardrails and spend limits. Most production systems end up using both.

### How is Workers AI priced?

By usage, in a unit Cloudflare calls Neurons, which measures the GPU compute a request consumed. The pricing page lists each model's rate per million input and output tokens, or per image or audio minute, next to the Neuron equivalent. There is a small free daily allocation and no charge for idle capacity.

### Does AI Gateway cost extra?

The core features, including analytics, caching and rate limiting, are free on all plans. You pay for what it uses underneath: stored logs under Workers Logs pricing on newer accounts, Guardrails as Workers AI inference, Logpush exports, and a percentage fee on credits if you choose Unified Billing over your own provider keys.

### Can AI Gateway cache similar prompts, not only identical ones?

Not at the time of writing. The documentation says caching applies only to identical requests: the key is built from the provider, endpoint, model, authentication header and the full request body. You can set a custom cache key per request. Semantic caching is described as planned, so do not put it in a budget yet.
