What They Actually Save (and How to Measure It Yourself)

LLM Gateways in 2026: What They Actually Save (and How to Measure It Yourself)

SitePoint TeamPublished inAI·Programming·
August 6, 2026

The AI briefing for Developers

Stay up to date with AI tools, model releases, and developer workflows that matter.

Weekly. Free. One click to leave.

SitePoint Premium
Stay Relevant and Grow Your Career in Tech

• Premium Results
• Publish articles on SitePoint
• Daily curated jobs
• Learning Paths
• Discounts to dev tools

7 Day Free Trial. Cancel Anytime.

If your application calls more than one model provider, you have probably already written the glue: a wrapper that swaps an OpenAI client for an Anthropic one, a retry loop for when a provider 500s, a spreadsheet where someone reconciles three invoices at the end of the month.

An LLM gateway replaces that glue with a single endpoint. In 2026 it is increasingly treated as standard infrastructure rather than an optimization — the layer where cost, reliability, and governance are actually enforced.

This guide covers what a gateway does, how the main options differ, and — the part most comparisons skip — which savings claims hold up on real traffic and how to check yours before committing.

Table of Contents

What Is an LLM Gateway?

An LLM gateway is a proxy that sits between your application and the model providers it calls. Instead of your code speaking OpenAI’s dialect to OpenAI, Anthropic’s to Anthropic, and Google’s to Google, it speaks one dialect to the gateway, and the gateway handles the rest.

Most gateways own six concerns:

1. Provider abstraction — one API surface, usually OpenAI-compatible, for every model behind it.
2. Routing — choosing which model serves a given request, by cost, latency, or difficulty.
3. Failover and retries — when a provider degrades, requests move rather than fail.
4. Caching — reusing work instead of paying for it twice.
5. Observability — per-request logs, latency distributions, and spend attribution by team or feature.
6. Rate limits and governance — budget caps, key management, and access control.

The abstraction layer is the part everyone agrees on. The other five are where products diverge, and where the marketing gets loose.

Do You Actually Need One?

Not everyone does. A gateway is worth it when at least one of these is true:

• You call more than one provider, or expect to. Provider abstraction alone justifies it.
• A provider outage would be a customer-visible incident. Failover is hard to retrofit under pressure.
• You cannot answer “which feature spent the most last month?” Attribution is the most underrated reason to adopt one.
• Your spend is large enough that a percentage matters. Below roughly $200–300/month in API spend, most gateway savings are smaller than the time you’d spend configuring them.

If you are a single developer calling one provider with a few thousand requests a month, skip it. The complexity is real and the savings are not.

The Main Options Compared

On the latency benchmarks you’ll find elsewhere: most published gateway latency comparisons come from gateway vendors, and the numbers vary by an order of magnitude betweenr own infrastructure to the gateway you are considering. Self-hosted proxies in your own VPC will generally beat a hosted hop across the public internet; beyond that, measure it

What Gateways Actually Save

This is where you should be skeptical, including of the vendors in the table above. The mechanisms are real but they have very different magnitudes, and each one only pays on a specific traffic shape.

Model Routing: Real, But Increasingly Commoditized

Routing sends easy requests to cheap models and hard ones to expensive models. On suitable traffic the price delta is large — a frontier model to its mini variant is often 5–10x.

Two caveats. First, routing only fires on requests that are short, tool-free, and not already structured-output — which excludes most agentic traffic. Second, this is no longer a differentiator you should pay for on its own. OpenRouter’s automatic router does model selection at no additional cost. Any gateway charging a premium purely for routing is selling something available for free.

Response Caching: The Highest-Leverage Mechanism, on the Right Traffic

Two different things get called “caching,” and the distinction is worth money.

Provider prompt caching (offered natively by OpenAI and Anthropic) discounts the repeated prefix of a request — typically around 90% off those input tokens. It is excellent, and you do not need a gateway to get it. What you do need is to not break it, which brings us to the most common own-goal in this category: rewriting or compressing conversation history invalidates the prefix and destroys the discount you were already receiving.

Response caching is different: if an identical request has been seen recently, the gateway returns the stored answer and never calls a model at all. That is not a discount on the request, it is the entire cost of the request. On traffic where the same requests genuinely recur, nothing else on this list comes close.

The catch is that “identical” is strict. Byte-identical repeats are common in classification pipelines, catalog enrichment, evaluation suites in CI, scheduled jobs, and retry storms. They are rare in interactive chat, where phrasing varies constantly.

Some gateways offer semantic caching — serving a cached answer for a request that is merely similar — which dramatically raises hit rates.

Treat semantic caching carefully. Similarity is not equivalence, and the failure mode is silent: “Is Python faster than Rust?” and “Is Rust faster than Python?” embed almost identically while having opposite answers. If you enable it, restrict it to prompt classes where a near-match genuinely implies the same answer, and log what it would have served before you let it serve anything.

The gateways that take response caching seriously tend to build the whole product around it rather than bolting it on. SlashSpend, for example, sticks to exact-match caching only — it deliberately leaves semantic matching out rather than ship the false-positive risk described above — and pairs it with a per-request decision receipt showing whether a given call was served from cache, downgraded to a cheaper model, compressed, or passed through untouched. That receipt-per-request model is worth looking for in any gateway you evaluate: without it, “our customers save 40%” is a claim you can’t audit down to a single request.

Prompt Compression: Small, and Easy to Get Wrong

Trimming filler from prompts saves input tokens. The savings are real but usually low single-digit percentages, and — as above — compressing anything but the newest message risks costing you more in lost prefix-cache discounts than it saves in tokens. Worth doing carefully; not worth choosing a gateway over.

The Finding That Is Usually Bigger Than All of Them

Before optimizing anything, check whether your own system prompt contains a rotating fragment.

Injecting a timestamp, request ID, or session ID into a system prompt is extremely common — several popular frameworks do it by default — and it silently busts your provider’s prefix cache on every single call. If you have a 2,000-token system prompt paying full price instead of the cached rate because of an injected Current time: line, that single bug can dwarf every gateway feature in this article combined.

It is also free to fix, and requires no gateway at all. Check this first.

Measure Your Own Traffic Before You Commit

Every number in the previous section depends on a property of your traffic: how often requests repeat, how many are simple enough to downgrade, whether your prefix is stable. Vendor case studies tell you what someone else’s traffic did.

You can measure yours locally before signing up for anything. slashspend-audit is an MIT-licensed CLI that reads request logs you already have and reports what routing, caching, and compression would have saved — running entirely on your machine, with no account and no upload:

npx slashspend-audit path/to/your-logs.jsonl

It reads OpenAI-shape JSONL, Claude Code session transcripts, Codex rollout files, and provider usage exports. If you don’t have request logs — and most teams don’t, because billing exports contain no prompt content — it ships a local capture proxy that records a representative sample:

npx slashspend-audit capture exportOPENAI_BASE_URL=http://localhost:8787

The output is a per-mechanism breakdown: how many requests a difficulty router would have downgraded and how many it correctly declines, how many exact repeats fell inside a cache window, what compression would have saved, and whether you have the rotating-prefix bug described above.

Two things worth knowing about it. It reports declines as prominently as savings — if your traffic is agentic, expect it to tell you routing does almost nothing, which is the honest answer. And it deliberately will not tell you whether a downgraded model’s answer would have been as good; that is a harder problem than cost accounting, and every run prints a “what this cannot tell you” section rather than pretending otherwise.

Run it before you believe anyone’s savings claim, including the ones on this page.

If the numbers look good on your own traffic, SlashSpend runs the identical routing, caching, and compression logic live, against production requests, instead of after the fact — the same decision code the audit CLI imports, with the receipt for every call available at /v1/trace so the number stays checkable after you’re paying for it, not just before.

Build or Buy?

Building a gateway is deceptively easy to start and expensive to finish. A day gets you provider abstraction. What takes months is everything after: streaming edge cases, per-provider retry semantics, token accounting that matches the invoice, cache invalidation, and the observability you’ll want at 3am.

Build if the gateway is genuinely core to your product, or if regulatory constraints prevent a third party in the request path — and note that self-hosting LiteLLM satisfies that second condition without writing anything.

Otherwise, buy. For most teams in 2026 the question is which gateway, not whether.

How to Choose

1. Can a third party be in your request path? If no, self-host LiteLLM and stop here.
2. Do you need breadth of models more than anything else? OpenRouter, and its free automatic router covers routing.
3. Is observability and governance the actual pain? Portkey.
4. Is your traffic highly repetitive — classification, enrichment, scheduled jobs, CI evaluations? Response caching is your biggest lever; measure your repeat rate first, then pick a gateway that caches responses rather than just proxying them.
5. Are you mostly running interactive or agentic traffic? Expect modest gateway savings, and prioritize abstraction, failover, and attribution over cost features.

Frequently Asked Questions

What is the difference between an LLM gateway and an LLM router?

A router chooses which model handles a request. A gateway is the broader layer — routing is one of its functions, alongside abstraction, caching, failover, observability, and rate limiting. Most products sold as routers today are gateways.

Does an LLM gateway add latency?

Yes, some. A self-hosted proxy inside your own network adds very little; a hosted gateway adds a network hop. When a request is served from a response cache, the gateway is dramatically faster than calling a model. Measure against your own baseline rather than trusting published figures.

Can a gateway break provider prompt caching?

It can, and this is the most common way teams lose money while trying to save it. Any transformation that alters the stable prefix of a request — rewriting history, reordering messages, aggressive compression — invalidates the provider’s prefix cache. Verify your cache-hit rate before and after adopting one.

Is an LLM gateway worth it for a small project?

Usually not for cost reasons alone. Below a few hundred dollars a month, gateway savings are smaller than the setup time. Adopt one for provider abstraction or failover if you need those; don’t adopt one expecting a meaningful bill reduction.

Do I need to give a gateway my API keys?

It depends on the product. Hosted aggregators like OpenRouter bill you directly and manage provider relationships. BYOK gateways use your own keys — check specifically whether keys are stored at rest or held only in memory, since the answer varies and materially changes your exposure. SlashSpend’s BYOK model, for instance, holds keys in memory for the duration of a request rather than persisting them — worth confirming as the specific pattern, since “BYOK” alone doesn’t tell you which one a given gateway uses.

Further Reading

• Prompt Compression and Cache Tuning: Cut Your LLM API Costs by 60% — the mechanics behind the compression and cache-tuning trade-offs above
• Context Compression Techniques: Reduce LLM Costs by 50% — token-level optimization before the gateway layer
• Local LLMs vs Cloud APIs: 2026 Total Cost of Ownership Analysis — when the answer is “don’t call an API at all”
• Self-Hosted LLM Costs 2026 — the infrastructure side of the same decision
• Open-Source vs Commercial LLMs: The Complete Guide (2026) — choosing what sits behind the gateway
• Headroom: The Token-Compression CLI That Cuts Your LLM API Costs — a complementary tool for the compression half

Sharing our passion for building incredible internet things.