Cut Node.js LLM Costs

Prompt Caching API Example: Cut Token Costs in Node.js

SitePoint TeamPublished inAI·Programming·APIs·
October 21, 2026

The AI briefing for Developers

Stay up to date with AI tools, model releases, and developer workflows that matter.

Weekly. Free. One click to leave.

SitePoint Premium
Stay Relevant and Grow Your Career in Tech

• Premium Results
• Publish articles on SitePoint
• Daily curated jobs
• Learning Paths
• Discounts to dev tools

7 Day Free Trial. Cancel Anytime.

Prompt caching reduces LLM token costs by reusing a previously processed prompt prefix so the provider skips the need to re-tokenize and re-encode it on subsequent requests. This tutorial walks through building a TypeScript request-builder and usage-logging helper that normalizes prompt caching behavior across all three providers in Node.js.

Table of Contents

Why Prompt Caching Matters for Node.js Developers

Prompt caching reduces LLM token costs by reusing a previously processed prompt prefix so the provider skips the need to re-tokenize and re-encode it on subsequent requests. For Node.js developers making repeated API calls with stable system instructions, few-shot examples, or long reference documents, prompt caching can deliver up to 90% cost reduction on cached tokens.

The problem is that three major providers — OpenAI, Anthropic, and Gemini — each implement prompt caching differently. Opt-in mechanisms, minimum token thresholds, TTL behavior, cache-hit indicators, and discount rates all vary. Developers must piece together incompatible documentation.

The result is a single factory function that constructs cache-optimized requests and a unified logger that verifies cache hits regardless of provider.

This tutorial walks through building a TypeScript request-builder and usage-logging helper that normalizes prompt caching behavior across all three providers in Node.js. The result is a single factory function that constructs cache-optimized requests and a unified logger that verifies cache hits regardless of provider.

Prerequisites: Node.js 20+, TypeScript 5+, an API key for at least one of the three providers, and familiarity with async/await and REST API consumption.

How Prompt Caching Works Across Providers

The Shared Principle: Stable Prompt Prefixes

All three providers cache from the beginning of the prompt forward — only the prefix qualifies. A “stable prefix” is any content that does not change between requests: system instructions, few-shot examples, long reference documents, or boilerplate context. Keep the prefix byte-identical across calls. Any mutation, even a whitespace change, invalidates the cache and forces a full re-processing pass.

Prompt design matters. Front-load all static content and append variable content (the user’s actual question, dynamic context) at the end.

Provider-Specific Caching Rules

The following table captures the critical differences across OpenAI, Anthropic, and Gemini:

For explicit caching, Gemini requires the largest minimum prefix at 32,768 tokens, making it suitable only for very large context windows. Gemini 2.5 and newer also enable implicit caching by default, which works more like OpenAI’s automatic approach. Anthropic offers the steepest read-token discount at 90%, though Anthropic bills cache creation calls at 25% above standard input pricing. OpenAI is zero-config but offers the smallest discount at 50%.

Initializing the Node.js Project

mkdir prompt-cache-demo &&cd prompt-cache-demonpm init -ynpminstall openai@^4.0.0 @anthropic-ai/sdk@^0.24.0 @google/generative-ai@^0.21.0 dotenv@^16.4.0npminstall-D typescript @types/nodenpx tsc --init

Update tsconfig.json with these essentials:

{"compilerOptions":{"target":"ES2022","module":"NodeNext","moduleResolution":"NodeNext","outDir":"./dist","rootDir":"./src","strict":true,"esModuleInterop":true},"include":["src"]}

OPENAI_API_KEY=your_openai_api_key_hereANTHROPIC_API_KEY=your_anthropic_api_key_hereGEMINI_API_KEY=your_gemini_api_key_here

Defining Shared TypeScript Interfaces

exporttypeProvider="openai"|"anthropic"|"gemini";exportinterfaceCacheAwareRequest{systemPrompt:string;userMessage:string;provider: Provider;}exportinterfaceCacheUsageReport{totalTokens:number;cachedTokens:number;cacheHit:boolean;provider: Provider;cachedTokenDiscountPercent:number;estimatedCostSavingsPercent:number;}

The CacheAwareRequest holds the provider-agnostic input shape. The CacheUsageReport normalizes the output so that calling code never needs to know which provider-specific field indicates a cache hit. The cachedTokenDiscountPercent reflects the provider’s discount rate on cached tokens, while estimatedCostSavingsPercent approximates actual savings relative to total tokens.

Building the TypeScript Request-Builder

Structuring Stable Prefixes for OpenAI

OpenAI caches automatically. There is no flag to set, no rest be at least 1,024 tokens long. The system message is the natural place for the stable prefix

import"dotenv/config";import OpenAI from"openai";importtype{ ChatCompletionCreateParams }from"openai/resources/chat/completions";const client =newOpenAI();exportasyncfunctionbuildOpenAIRequest(systemPrompt:string,userMessage:string){const params: ChatCompletionCreateParams ={model:"gpt-4o",messages:[{ role:"system", content: systemPrompt },{ role:"user", content: userMessage },],};try{const response =await client.chat.completions.create(params);return response;}catch(err){const message = err instanceofError? err.message :String(err);thrownewError(`OpenAI API request failed:${message}`,{ cause: err });}}

The systemPrompt string should contain the full stable prefix. As long as it exceeds 1,024 tokens and remains identical across calls, OpenAI will cache it automatically within its TTL window.

Structuring Stable Prefixes for Anthropic

Anthropic requires explicit opt-inlast content block of the stable prefix. Everything before (and including) that block is cached; everything after is processed fresh

import"dotenv/config";import Anthropic from"@anthropic-ai/sdk";const client =newAnthropic();exportasyncfunctionbuildAnthropicRequest(systemPrompt:string,userMessage:string){try{const response =await client.messages.create({model:"claude-3-5-sonnet-20241022",max_tokens:1024,system:[{type:"text",text: systemPrompt,cache_control:{ type:"ephemeral"},},],messages:[{ role:"user", content: userMessage }],});return response;}catch(err){const message = err instanceofError? err.message :String(err);thrownewError(`Anthropic API request failed:${message}`,{ cause: err });}}

The cache_control: { type: "ephemeral" } annotation marks the boundary. On the first call, Anthropic reports cache_creation_input_tokens. On subsequent calls within the 5-minute TTL, it reports cache_read_input_tokens instead.

Structuring Stable Prefixes for Gemini

Gemini uses a two-step process. First, create a cachedContent recit TTL. Then reference that re2,768 tokens

Note: The GoogleAICacheManager class may be exported from @google/generative-ai/server rather than the top-level @google/generative-ai depending on your installed SDK version. If the import below fails, try import { GoogleAICacheManager } from "@google/generative-ai/server". Verify with: node -e "const m = require('@google/generative-ai'); console.log(typeof m.GoogleAICacheManager)" — if it prints undefined, use the /server subpath.

import"dotenv/config";import{GoogleGenerativeAI,GoogleAICacheManager,}from"@google/generative-ai";importtype{ CachedContent }from"@google/generative-ai";constGEMINI_API_KEY= process.env.GEMINI_API_KEY;if(!GEMINI_API_KEY){thrownewError("GEMINI_API_KEY environment variable is not set. "+"Add it to your .env file before importing this module.");}const genAI =newGoogleGenerativeAI(GEMINI_API_KEY);const cacheManager =newGoogleAICacheManager(GEMINI_API_KEY);exportasyncfunctionbuildGeminiCachedContent(systemPrompt:string){try{const cachedContent =await cacheManager.create({model:"models/gemini-1.5-pro-001",contents:[{role:"user",parts:[{ text: systemPrompt }],},],ttlSeconds:3600,});return cachedContent;}catch(err){const message = err instanceofError? err.message :String(err);thrownewError(`Gemini cache creation failed:${message}`,{ cause: err });}}exportasyncfunctionbuildGeminiRequest(cachedContent: CachedContent,userMessage:string){try{const model = genAI.getGenerativeModelFromCachedContent(cachedContent);const result =await model.generateContent(userMessage);return result.response;}catch(err){const message = err instanceofError? err.message :String(err);thrownewError(`Gemini API request failed:${message}`,{ cause: err });}}

The cached content object must be stored and reused. If the TTL expires, the re name to a database or cache store to survive process restarts; the in-memory Map used in the factory below is lost on every cold start, causing redundant cachedContent creation and additional billing

Unifying with a Request-Builder Factory

A single factory function dispatches to the correct provider-specific builder and returns a normalized shape alongside the raw response for usage inspection.

import{ createHash }from"node:crypto";import{ buildOpenAIRequest }from"./builders/openai.js";import{ buildAnthropicRequest }from"./builders/anthropic.js";import{buildGeminiCachedContent,buildGeminiRequest,}from"./builders/gemini.js";importtype{ Provider }from"./types.js";importtype{ CachedContent }from"@google/generative-ai";const geminiCacheObjects =newMap<string, CachedContent>();const geminiCacheInflight =newMap<string,Promise<CachedContent>>();exportasyncfunctionbuildCachedRequest(provider: Provider,systemPrompt:string,userMessage:string){switch(provider){case"openai":{const response =awaitbuildOpenAIRequest(systemPrompt, userMessage);return{ text: response.choices[0]?.message?.content ??"", raw: response };}case"anthropic":{const response =awaitbuildAnthropicRequest(systemPrompt, userMessage);const textBlock = response.content.find((b)=> b.type ==="text");return{ text: textBlock?.text ??"", raw: response };}case"gemini":{const cacheKey =createHash("sha256").update(systemPrompt).digest("hex");if(!geminiCacheObjects.has(cacheKey)){if(!geminiCacheInflight.has(cacheKey)){const inflight =buildGeminiCachedContent(systemPrompt).then((cached)=>{geminiCacheObjects.set(cacheKey, cached);geminiCacheInflight.delete(cacheKey);return cached;});geminiCacheInflight.set(cacheKey, inflight);}await geminiCacheInflight.get(cacheKey)!;}const response =awaitbuildGeminiRequest(geminiCacheObjects.get(cacheKey)!,userMessage);const candidate = response.candidates?.[0];const text =candidate?.content?.parts?.map((p:{ text?:string})=> p.text ??"").join("")??"";return{ text, raw: response };}default:{const _exhaustive:never= provider;thrownewError(`Unsupported provider: "${_exhaustive}".`+`Valid values are: "openai" | "anthropic" | "gemini".`);}}}

The Gemini path stores cached content objects in a Map keyed by a SHA-256 hash of the full system prompt, avoiding both redundant cache creation on repeated calls and cache key collisions that could occur with prefix-based keys. A secondary inflight Map ensures that concurrent requests with the same prompt do not trigger duplicate cache creation calls.

Verifying Cache Hits with a Usage-Logging Helper

Parsing OpenAI Usage Metrics

OpenAI exposes cache data at response.usage.prompt_tokens_details.cached_tokens. When this value exceeds zero, the request hit the cache.

Parsing Anthropic Usage Metrics

Anthropic splits caching into two fields: response.usage.cache_creation_input_tokens (populated on the first request when the provider writes the cache) and response.usage.cache_read_input_tokens (populated on subsequent requests that read from it). Check whether cache_read_input_tokens > 0 to confirm a hit.

Parsing Gemini Usage Metrics

Check response.usageMetadata.cachedContentTokenCount. If it exceeds zero, the cached content re

The Unified logCacheUsage Helper

import{ createHash }from"node:crypto";importtype{ Provider, CacheUsageReport }from"./types.js";constDISCOUNT_RATES: Record<Provider,number>={openai:50,anthropic:90,gemini:75,};constPROVIDER_MINIMUMS: Record<Provider,number>={openai:1024,anthropic:1024,gemini:32768,};interfaceOpenAIRaw{usage?:{total_tokens?:number;prompt_tokens_details?:{ cached_tokens?:number};};}interfaceAnthropicRaw{usage?:{input_tokens?:number;output_tokens?:number;cache_read_input_tokens?:number;};}interfaceGeminiRaw{usageMetadata?:{promptTokenCount?:number;candidatesTokenCount?:number;cachedContentTokenCount?:number;};}typeRawResponse= OpenAIRaw | AnthropicRaw | GeminiRaw;exportfunctionlogCacheUsage(raw: RawResponse, provider: Provider): CacheUsageReport {let totalTokens =0;let cachedTokens =0;switch(provider){case"openai":{const r = raw as OpenAIRaw;totalTokens = r.usage?.total_tokens ??0;cachedTokens = r.usage?.prompt_tokens_details?.cached_tokens ??0;break;}case"anthropic":{const r = raw as AnthropicRaw;totalTokens =(r.usage?.input_tokens ??0)+(r.usage?.output_tokens ??0);cachedTokens = r.usage?.cache_read_input_tokens ??0;break;}case"gemini":{const r = raw as GeminiRaw;totalTokens =(r.usageMetadata?.promptTokenCount ??0)+(r.usageMetadata?.candidatesTokenCount ??0);cachedTokens = r.usageMetadata?.cachedContentTokenCount ??0;break;}}const cacheHit = cachedTokens >0;const cachedTokenDiscountPercent = cacheHit ?DISCOUNT_RATES[provider]:0;const estimatedCostSavingsPercent =totalTokens >0&& cacheHit?(cachedTokens / totalTokens)*DISCOUNT_RATES[provider]:0;const report: CacheUsageReport ={totalTokens,cachedTokens,cacheHit,provider,cachedTokenDiscountPercent,estimatedCostSavingsPercent,};console.table(report);return report;}exportfunctionlogCacheUsageDebug(raw: RawResponse,provider: Provider,systemPrompt:string): CacheUsageReport {const report =logCacheUsage(raw, provider);const prefixHash =createHash("sha256").update(systemPrompt).digest("hex").slice(0,16);const estimatedTokens = Math.ceil(systemPrompt.split(/s+/).length *1.3);const minimum =PROVIDER_MINIMUMS[provider];console.log(`[DEBUG] Prefix hash:${prefixHash}`);console.log(`[DEBUG] Estimated prefix tokens:${estimatedTokens}`);console.log(`[DEBUG] Provider minimum:${minimum}`);console.log(`[DEBUG] Meets minimum:${estimatedTokens >= minimum}`);console.log(`[DEBUG] Timestamp:${newDate().toISOString()}`);if(!report.cacheHit && estimatedTokens < minimum){console.warn(`[WARN] Prefix below${provider}minimum of${minimum}tokens`);}return report;}

The cachedTokenDiscountPercent reflects the provider’s discount rate applied to cached tokens specifically. The estimatedCostSavingsPercent approximates the actual savings relative to total tokens in the request — for example, if only 100 of 10,000 tokens were cached at Anthropic’s 90% rate, the estimated savings would be ~0.9%, not 90%. The hash comparison across requests reveals whether the prefix changed. The token estimate uses a 1.3x multiplier on word count as a rough approximation (+-30% error); production systems should use provider-specific tokenizers (e.g., tiktoken for OpenAI) for precision.

Inspecting Cache Misses and Debugging

Common Reasons for Cache Misses

Cache misses happen for several concrete reasons. You changed the prefix between requests (even whitespace or trailing newline differences count). Your prefix falls below the provider’s minimum token threshold. The TTL expired — Anthropic’s 5-minute default is particularly aggressive. You switched the model parameter between requests. Or, in Gemini’s case, you did not pass the cachedContent name to the generation call.

End-to-End Demo: Running Two Consecutive Requests

import"dotenv/config";import{ buildCachedRequest }from"./requestBuilder.js";import{ logCacheUsageDebug }from"./cacheLogger.js";importtype{ Provider }from"./types.js";constVALID_PROVIDERS: Provider[]=["openai","anthropic","gemini"];const rawProvider = process.argv[2]?? process.env.PROVIDER??"openai";if(!VALID_PROVIDERS.includes(rawProvider as Provider)){thrownewError(`Invalid provider: "${rawProvider}". Valid values are:${VALID_PROVIDERS.join(", ")}`);}const provider: Provider = rawProvider as Provider;const systemPrompt =`You are an expert legal analyst. Below is the full text of the terms of service for analysis:${"Section: The following terms govern use of the platform and all associated services. Users must comply with all applicable regulations and guidelines as specified herein. ".repeat(60)}`;const userMessage ="Summarize the key obligations for end users.";asyncfunctionmain(){console.log(`--- Request 1 (${provider}): Expect cache MISS ---`);const result1 =awaitbuildCachedRequest(provider, systemPrompt, userMessage);logCacheUsageDebug(result1.raw, provider, systemPrompt);if(provider ==="openai"){awaitnewPromise((r)=>setTimeout(r,2000));}console.log(`--- Request 2 (${provider}): Expect cache HIT ---`);const result2 =awaitbuildCachedRequest(provider, systemPrompt, userMessage);logCacheUsageDebug(result2.raw, provider, systemPrompt);}main().catch(console.error);

Run with npx tsc && node dist/demo.js openai. Swap the argument to anthropic or gemini to test other providers. The first request populates the cache; the second reads from it.

Cost and Latency Impact: What to Expect

Realistic savings depend on reuse frequency within the TTL window. OpenAI provides a 50% discount on cached input tokens. Anthropic provides 90% on cached read tokens, though Anthropic bills cache creation calls at 25% above standard input pricing. Gemini provides 75% for Gemini 1.5 Pro (the Flash rate is not confirmed at time of writing). Anthropic reports reduced latency on cache hits, but none of the three providers publish fixed latency benchmarks for caching; consult each provider’s official prompt caching documentation for current figures.

These savings only materialize when the same prefix is reused frequently within the cache’s TTL. A prefix that changes on every request, or one that falls below the provider’s minimum token threshold, will never trigger caching, and you will pay standard pricing.

Next Steps

Start by adding streaming response support to the request-builder, since cached prefixes work with streaming on all three providers. From there, implement persistent TTL management for Gemini’s cachedContent re, and wire the cache usage logger into an observability pipeline like OpenTelemetry spans for production cost monitoring

For full details on each provider’s caching behavior, consult the official documentation: OpenAI Prompt Caching, Anthropic Prompt Caching, and Gemini Context Caching.

Sharing our passion for building incredible internet things.