Testing LLM Output in CI with Vitest and Schema Validation

Build Deterministic LLM Eval Suites in CI with Vitest and Zod

SitePoint TeamPublished inAI·Programming·JavaScript·
September 23, 2026

The AI briefing for Developers

Stay up to date with AI tools, model releases, and developer workflows that matter.

Weekly. Free. One click to leave.

SitePoint Premium
Stay Relevant and Grow Your Career in Tech

• Premium Results
• Publish articles on SitePoint
• Daily curated jobs
• Learning Paths
• Discounts to dev tools

7 Day Free Trial. Cancel Anytime.

How to Build Deterministic LLM Eval Suites With Vitest and Zod

1. Install Vitest, Zod, and the OpenAI SDK as project dependencies.
2. Record live LLM responses as JSON fixture files with model version and prompt hash metadata.
3. Pre-compute embedding vectors for reference and output texts, storing them alongside fixtures.
4. Build a ReplayProvider class that reads fixtures offline, replacing live API calls in tests.
5. Define Zod schema contracts that validate output structure, types, enums, and numeric ranges.
6. Create custom Vitest matchers for Levenshtein edit distance and cosine similarity assertions.
7. Configure a dedicated Vitest config targeting .eval.test.ts files with isolated timeouts and reporters.
8. Wire a GitHub Actions workflow to run evals on pull requests and nightly, caching fixtures and uploading reports.

Large language models produce different outputs for the same prompt across successive calls. For teams shipping generative AI features in TypeScript applications, this non-determinism breaks the fundamental assumption behind traditional unit tests. This guide walks through building a deterministic, zero-cost CI evaluation suite using Vitest, replay fixtures, and Zod schema contracts.

Table of Contents

Why LLM Output Needs Deterministic Testing

Large language models produce different outputs for the same prompt across successive calls. Temperature settings, model updates, and provider-side infrastructure changes all contribute to this variance. For teams shipping generative AI features in TypeScript applications, this non-determinism breaks the fundamental assumption behind traditional unit tests: that a given input produces a predictable output.

Standard equality assertions cannot handle free-text generation. Snapshot testing fares no better, since even minor wording changes cause false failures. Teams that want deterministic LLM tests in TypeScript typically evaluate Python-centric tools like promptfoo or DeepEval, which introduce cross-language dependencies, separate virtual environments, and unfamiliar configuration paradigms. For a full-stack TypeScript team already running Vitest in CI, maintaining a parallel Python virtualenv, a second CI step, and extra config files adds friction that rarely pays off.

“Deterministic” here does not mean freezing the model’s output into a static string. It means combining replay fixtures (recorded LLM responses replayed without network calls) with structural contracts (Zod schemas that validate shape, types, and constraints) and fuzzy similarity assertions (Levenshtein distance and cosine similarity checks that catch semantic drift without demanding character-perfect matches).

Together, these techniques create a prompt regression testing workflow that lives entirely within an existing TypeScript CI pipeline and costs nothing beyond the initial fixture recording.

Prerequisites

This article assumes the following tools and versions:

• Node.js ≥ 18
• vitest ^2.0 (the pool: "forks" and poolOptions.forks API used below requires Vitest ≥ 1.0)
• zod ^3.22
• openai ^4.0 (for the client.chat.completions.create API shape)
• OPENAI_API_KEY environment variable (required only for fixture recording, never during CI test runs)

Install the runtime dependencies:

npminstall vitest zod openai

Package manager note: This article uses npm. If your project uses pnpm or yarn, substitute accordingly (and update cache: "npm" in the GitHub Actions workflow to cache: "pnpm" or cache: "yarn").

Architecture of a TypeScript LLM Eval Harness

Core Components Overview

The harness relies on four interlocking components. Replay fixtures are JSON files containing previously recorded LLM responses, tagged with metadata such as model version, temperature, and recording timestamp. Zod schema contracts define the structural expectations for parsed outputs, catching regressions in field presence, types, enum values, and numeric ranges. Fuzzy matchers handle the free-text dimension: Levenshtein distance detects character-level drift, while cosine similarity over pre-computed embedding vectors detects semantic drift. Where the first three components run locally without external dependencies, the fourth component, a GitHub Actions workflow, ties them together by running eval tests on pull requests and on a nightly schedule.

Project Structure

Placing eval tests under eval/ with a dedicated glob prevents Vitest’s default config from running them, letting you apply different timeout and concurrency settings without affecting the rest of your test suite.

project-root/├── eval/│   ├── __fixtures__/│   │   ├── product-description-gpt4o.fixture.json│   │   └── embeddings/│   │       ├── product-description-reference.embedding.json│   │       └── product-description-output.embedding.json│   ├── schemas/│   │   └── product-description.schema.ts│   ├── matchers/│   │   ├── toBeWithinEditDistance.ts│   │   └── toBeSemanticallySimilar.ts│   ├── utils/│   │   ├── recordFixture.ts│   │   ├── recordEmbeddings.ts│   │   └── ReplayProvider.ts│   └── product-description.eval.test.ts├── vitest.config.eval.ts└── .github/└── workflows/└── llm-eval.yml

The __fixtures__/ directory holds recorded responses and cached embeddings. The schemas/ directory contains Zod definitions. The matchers/ directory houses custom Vitest assertion extensions. The utils/ directory provides the recording and replay infrastructure. Eval test files use the .eval.test.ts suffix so the dedicated Vitest config can target them with a specific glob pattern.

Recording and Replaying LLM Fixtures

Capturing Live Responses as Fixtures

The record-once, replay-always pattern ensures that the code calls the live API only when you explicitly record fixtures, never during CI test runs. Each fixture file includes the raw response alongside metadata that pins the exact conditions under which the response was generated.

import{ mkdir, writeFile }from"fs/promises";import path from"path";import{ createHash }from"crypto";import{ fileURLToPath }from"url";import{ dirname }from"path";import OpenAI from"openai";const __filename =fileURLToPath(import.meta.url);const __dirname =dirname(__filename);interfaceFixtureMeta{model:string;temperature:number;recordedAt:string;promptHash:string;}exportasyncfunctionrecordFixture(fixtureId:string,prompt:string,options:{ model:string; temperature:number}):Promise<void>{const fixturePath = path.resolve(__dirname,`../__fixtures__/${fixtureId}.fixture.json`);if(!process.env.OPENAI_API_KEY){thrownewError("OPENAI_API_KEY environment variable is required for fixture recording");}const client =newOpenAI({ apiKey: process.env.OPENAI_API_KEY});const response =await client.chat.completions.create({model: options.model,temperature: options.temperature,messages:[{ role:"user", content: prompt }],});const content = response.choices[0]?.message?.content ??"";const meta: FixtureMeta ={model: options.model,temperature: options.temperature,recordedAt:newDate().toISOString(),promptHash:createHash("sha256").update(prompt).digest("hex").slice(0,16),};awaitmkdir(path.dirname(fixturePath),{ recursive:true});try{awaitwriteFile(fixturePath,JSON.stringify({ meta, output: content },null,2),{ encoding:"utf-8", flag:"wx"});console.log(`Fixture "${fixtureId}" recorded.`);}catch(err:unknown){if(typeof err ==="object"&&err !==null&&(err as NodeJS.ErrnoException).code ==="EEXIST"){console.log(`Fixture "${fixtureId}" already exists. Skipping.`);return;}throw err;}}

The promptHash field is a truncated SHA-256 digest of the prompt bytes. If the prompt template changes, the hash no longer matches, signaling that the fixture should be re-recorded. The writeFile call uses the "wx" flag (exclusive create), which atomically fails if the file already exists, preventing both accidental overwrites and race conditions when multiple recording runs execute concurrently.

Recording Embedding Vectors

The cosine similarity matcher (described later) depends on embedding vectors you pre-compute and store as JSON files. The following utility records those embeddings alongside your response fixtures:

import{ mkdir, writeFile }from"fs/promises";import path from"path";import{ fileURLToPath }from"url";import{ dirname }from"path";import OpenAI from"openai";const __filename =fileURLToPath(import.meta.url);const __dirname =dirname(__filename);exportasyncfunctionrecordEmbedding(embeddingId:string,text:string,options:{ model?:string}={}):Promise<void>{if(!process.env.OPENAI_API_KEY){thrownewError("OPENAI_API_KEY environment variable is required for embedding recording");}const model = options.model ??"text-embedding-3-small";const client =newOpenAI({ apiKey: process.env.OPENAI_API_KEY});const response =await client.embeddings.create({model,input: text,});if(!response.data[0]){thrownewError(`No embedding returned for "${embeddingId}". The API response contained an empty data array.`);}const vector = response.data[0].embedding;const embeddingsDir = path.resolve(__dirname,"../__fixtures__/embeddings");awaitmkdir(embeddingsDir,{ recursive:true});awaitwriteFile(path.join(embeddingsDir,`${embeddingId}.embedding.json`),JSON.stringify(vector),"utf-8");console.log(`Embedding "${embeddingId}" recorded (${vector.length}dims).`);}

Run this utility for both the reference text and each output you want to compare. For example:

awaitrecordEmbedding("product-description-reference", referenceText);awaitrecordEmbedding("product-description-output", fixtureOutputText);

Building a Replay Provider

During test execution, your test setup replaces the live OpenAI client with a replay provider that reads fixture JSON and returns it as a typed response. This guarantees zero network calls in CI, provided you wire vi.mock (or constructor-based dependency injection) correctly. The replay mechanism is the sole enforcement point for offline execution.

import{ readFile }from"fs/promises";import path from"path";import{ fileURLToPath }from"url";import{ dirname }from"path";import{ z }from"zod";const __filename =fileURLToPath(import.meta.url);const __dirname =dirname(__filename);const FixtureDataSchema = z.object({meta: z.object({model: z.string(),temperature: z.number(),recordedAt: z.string(),promptHash: z.string(),}),output: z.string(),});exporttypeFixtureData= z.infer<typeof FixtureDataSchema>;exportclassReplayProvider{private fixturesDir:string;constructor(fixturesDir:string){this.fixturesDir = fixturesDir;}asyncgetCompletion(fixtureId:string):Promise<FixtureData>{const fixturePath = path.resolve(this.fixturesDir,`${fixtureId}.fixture.json`);const raw =awaitreadFile(fixturePath,"utf-8");const parsed:unknown=JSON.parse(raw);const result = FixtureDataSchema.safeParse(parsed);if(!result.success){thrownewError(`Fixture "${fixtureId}" failed schema validation:`+result.error.issues.map((i)=>`→${i.path.join(".")}:${i.message}`).join(""));}return result.data;}}

Vitest’s vi.mock wires this into the dependency graph:

import{ vi }from"vitest";import{ ReplayProvider }from"./utils/ReplayProvider";import path from"path";import{ fileURLToPath }from"url";import{ dirname }from"path";const __filename =fileURLToPath(import.meta.url);const __dirname =dirname(__filename);const replay =newReplayProvider(path.resolve(__dirname,"./__fixtures__"));constFIXTURE_MAP: Record<string,string>={"product-description":"product-description-gpt4o",};vi.mock("../src/llm/client",()=>({getLLMResponse:async(prompt:string)=>{const key = Object.keys(FIXTURE_MAP).find((k)=> prompt.includes(k));if(!key){thrownewError(`No fixture mapping for prompt:${prompt.slice(0,60)}…`);}const fixture =await replay.getCompletion(FIXTURE_MAP[key]);return fixture.output;},}));

Note: The mock path '../src/llm/client' must match your production module’s actual path relative to the test file. Adjust it to reflect your project’s directory structure. If your production getLLMResponse signature differs from the mock, use constructor-based dependency injection (passing the ReplayProvider directly to the service under test) instead of vi.mock to avoid coupling production code to test infrastructure. The FIXTURE_MAP lookup ensures different prompts resolve to different fixture files; extend it as you add new eval scenarios.

Validating Structure with Zod Schema Contracts

Defining Output Schemas

Schemas catch more regression types than snapshots. When an LLM’s structured output drops a required field, returns a string where a number is expected, or introduces an unexpected enum value, a Zod schema fails with a precise error path. Snapshots, by contrast, fail on any textual change, including benign rewording.

import{ z }from"zod";exportconst SentimentEnum = z.enum(["positive","neutral","negative"]);exportconst ProductDescriptionSchema = z.object({title: z.string().min(1).max(200),bullets: z.array(z.string().min(1)).min(3,"At least 3 bullet points required").max(10),sentiment: SentimentEnum,confidence: z.number().min(0,"Confidence must be >= 0").max(1,"Confidence must be <= 1"),tags: z.array(z.string()).optional(),});exporttypeProductDescription= z.infer<typeof ProductDescriptionSchema>;

This schema models a product description generator that returns a title, bullet points, a sentiment classification, a confidence score between 0 and 1, and optional tags. The z.enum constraint ensures the sentiment value belongs to a closed set. The z.number().min().max() range constraint catches confidence scores outside the expected bounds.

Writing Vitest Assertions with Zod

Using schema.safeParse() inside Vitest assertions produces granular failure messages that pinpoint exactly which field violated which constraint, far more actionable than a generic “output mismatch” error.

import{ describe, it, expect }from"vitest";import{ ReplayProvider }from"./utils/ReplayProvider";import{ ProductDescriptionSchema }from"./schemas/product-description.schema";importtype{ ZodIssue }from"zod";import path from"path";import{ fileURLToPath }from"url";import{ dirname }from"path";const __filename =fileURLToPath(import.meta.url);const __dirname =dirname(__filename);const replay =newReplayProvider(path.resolve(__dirname,"./__fixtures__"));functionformatZodErrors(issues: ZodIssue[]):string{return issues.map((issue)=>`→${issue.path.join(".")}:${issue.message}`).join("");}describe("Product Description LLM Eval",()=>{it("should produce a structurally valid product description",async()=>{const fixture =await replay.getCompletion("product-description-gpt4o");let parsed:unknown;try{parsed =JSON.parse(fixture.output);}catch{thrownewError(`Fixture "product-description-gpt4o" output is not valid JSON.`+`Raw output (first 200 chars):${fixture.output.slice(0,200)}`);}const result = ProductDescriptionSchema.safeParse(parsed);if(!result.success){thrownewError(`Zod schema validation failed:${formatZodErrors(result.error.issues)}`);}expect(result.success).toBe(true);});it("should contain at least 3 bullet points",async()=>{const fixture =await replay.getCompletion("product-description-gpt4o");let parsed:unknown;try{parsed =JSON.parse(fixture.output);}catch{thrownewError(`Fixture "product-description-gpt4o" output is not valid JSON.`+`Raw output (first 200 chars):${fixture.output.slice(0,200)}`);}const result = ProductDescriptionSchema.safeParse(parsed);expect(result.success).toBe(true);if(result.success){expect(result.data.bullets.length).toBeGreaterThanOrEqual(3);}});});

The formatZodErrors utility translates ZodError.issues into a readable list with dotted paths. A failure in CI might render as → confidence: Confidence must be <= 1, making the regression immediately identifiable without reading raw Zod internals.

Fuzzy Similarity Assertions for Semantic Correctness

Levenshtein Distance for Near-Match Strings

Structural validation confirms the shape of the output. Fuzzy similarity assertions confirm the content has not drifted beyond an acceptable threshold. For short, predictable text fields like product titles, normalized Levenshtein distance requires no external dependencies and runs in O(n*m) time, making it practical for strings under a few hundred characters.

Choosing a threshold requires balancing sensitivity against noise. A normalized distance ratio of 0.15 (meaning up to 15% of characters may differ) works well for titles and short descriptions where minor wording changes are acceptable but wholesale rewrites should be flagged. Thresholds are more meaningful for strings of similar length; calibrate against your specific field lengths. Tighter thresholds like 0.05 are appropriate for highly constrained outputs; looser thresholds above 0.25 rarely catch meaningful regressions.

import{ expect }from"vitest";declaremodule"vitest"{interfaceAssertion<R=any>{toBeWithinEditDistance(reference:string, maxRatio?:number):R;}interfaceAsymmetricMatchersContaining{toBeWithinEditDistance(reference:string, maxRatio?:number):any;}}exportfunctionlevenshtein(a:string, b:string):number{if(a.length ===0)return b.length;if(b.length ===0)return a.length;let prev =Array.from({ length: b.length +1},(_, j)=> j);let curr =newArray<number>(b.length +1);for(let i =1; i <= a.length; i++){curr[0]= i;for(let j =1; j <= b.length; j++){curr[j]= Math.min(prev[j]+1,curr[j -1]+1,prev[j -1]+(a[i -1]=== b[j -1]?0:1));}[prev, curr]=[curr, prev];}return prev[b.length];}expect.extend({toBeWithinEditDistance(received:string,reference:string,maxRatio:number=0.15){const distance =levenshtein(received, reference);const maxLen = Math.max(received.length, reference.length);const ratio = maxLen ===0?0: distance / maxLen;const pass = ratio <= maxRatio;return{pass,message:()=>`Expected edit distance ratio ≤${maxRatio}, got${ratio.toFixed(4)}(distance:${distance}, max length:${maxLen})`,};},});

This custom matcher normalizes the raw Levenshtein distance by the length of the longer string, producing a ratio between 0 and 1. Tests invoke it as expect(output.title).toBeWithinEditDistance(referenceTitle, 0.15).

Embedding Cosine Similarity for Semantic Drift

For longer free-text content where character-level comparison is too brittle, cosine similarity over embedding vectors captures whether the meaning has drifted, not just the wording. To avoid live API calls in CI, you pre-compute embedding vectors when you record fixtures using the recordEmbeddings.ts utility shown earlier, then store them as JSON files alongside the response fixtures.

A threshold of 0.92, when using OpenAI’s text-embedding-3-small model, flags wholesale topic changes while passing single-sentence rephrasings. Recalibrate this threshold if you use a different embedding model, as similarity score distributions vary across models. Start at 0.92 and adjust after reviewing false positives and false negatives against your own prompt set.

import{ expect }from"vitest";declaremodule"vitest"{interfaceAssertion<R=any>{toBeSemanticallySimilar(referenceVector:number[],threshold?:number):R;}interfaceAsymmetricMatchersContaining{toBeSemanticallySimilar(referenceVector:number[],threshold?:number):any;}}exportfunctioncosineSimilarity(a:number[], b:number[]):number{if(a.length !== b.length){thrownewError(`Vector length mismatch: a=${a.length}, b=${b.length}`);}if(a.length ===0){thrownewError("Cannot compute cosine similarity of zero-length vectors");}let dot =0;let normA =0;let normB =0;for(let i =0; i < a.length; i++){dot += a[i]* b[i];normA += a[i]* a[i];normB += b[i]* b[i];}const denom = Math.sqrt(normA)* Math.sqrt(normB);if(denom ===0){thrownewError("Cannot compute cosine similarity: one or both vectors are zero");}return dot / denom;}expect.extend({toBeSemanticallySimilar(receivedVector:number[],referenceVector:number[],threshold:number=0.92){const similarity =cosineSimilarity(receivedVector, referenceVector);const pass = similarity >= threshold;return{pass,message:()=>`Expected cosine similarity ≥${threshold}, got${similarity.toFixed(6)}`,};},});

The matcher accepts pre-loaded embedding vectors directly, keeping the test execution entirely offline and avoiding synchronous file I/O inside the matcher body. Callers load vectors asynchronously before invoking the assertion:

import{ readFile }from"fs/promises";import path from"path";const referenceVector:number[]=JSON.parse(awaitreadFile(path.resolve(__dirname,"./__fixtures__/embeddings/product-description-reference.embedding.json"),"utf-8"));const outputVector:number[]=JSON.parse(awaitreadFile(path.resolve(__dirname,"./__fixtures__/embeddings/product-description-output.embedding.json"),"utf-8"));expect(outputVector).toBeSemanticallySimilar(referenceVector,0.92);

Wiring It All Into GitHub Actions CI

Vitest Configuration for Eval Suites

A dedicated Vitest configuration file isolates eval suite settings from the rest of the test infrastructure, allowing different timeouts, concurrency, and reporters.

import{ defineConfig }from"vitest/config";exportdefaultdefineConfig({test:{include:["eval/**/*.eval.test.ts"],testTimeout:30_000,pool:"forks",poolOptions:{forks:{singleFork:true,},},reporters:["default","json"],outputFile:{json:"./eval-report.json",},setupFiles:["./eval/matchers/toBeWithinEditDistance.ts","./eval/matchers/toBeSemanticallySimilar.ts",],},});

The include glob targets only .eval.test.ts files. singleFork: true runs all tests in a single worker process, eliminating cross-worker parallelism. It does not serialize async operations within a file; use test.sequential if strict ordering is required. The 30-second timeout provides headroom for large fixture files; typical runs complete in under 2 seconds, but the margin avoids flaky failures on resource-constrained CI runners. The JSON reporter produces a machine-readable report that can be uploaded as a CI artifact.

GitHub Actions Workflow Definition

Warning: Without a live re-recording step, the nightly run only replays cached fixtures. It cannot detect model-side changes. To enable actual drift detection, add a conditional step that re-records fixtures when a LIVE_EVAL secret is set, then commits the changes.

The workflow runs on pull requests for immediate feedback and on a nightly cron schedule.

name: LLM Eval Suiteon:pull_request:paths:-"eval/**"-"src/llm/**"-"vitest.config.eval.ts"schedule:-cron:"0 3 * * *"jobs:llm-eval:runs-on: ubuntu-lateststeps:-uses: actions/checkout@v4-uses: actions/setup-node@v4with:node-version:20cache:"npm"-name: Install dependenciesrun: npm ci-name: Restore fixture cacheuses: actions/cache@v4with:path: eval/__fixtures__key: llm-fixtures-${{ hashFiles('eval/__fixtures__/**', 'package-lock.json') }}restore-keys:|llm-fixtures--name: Run LLM eval suiterun: npx vitest run --config vitest.config.eval.ts-name: Upload eval reportif: always()uses: actions/upload-artifact@v4with:name: llm-eval-reportpath: eval-report.jsonretention-days:30

Note: The OPENAI_API_KEY must be configured as a GitHub Actions secret if you add a live re-recording step to the nightly workflow. The actions/*@v4 versions are current as of mid-2024; check the GitHub Actions marketplace for newer releases.

The paths filter on pull requests ensures GitHub Actions runs the eval suite only when relevant files change, conserving CI minutes. The fixture cache key is based on fixture content and the lockfile, so it invalidates when fixtures or dependencies change rather than when unrelated TypeScript condition on artifact upload ensures the JSON report is available even when tests fail, which is critical for debugging regressions

Putting It All Together: The Complete Eval Harness

The data flow through the harness follows a linear path:

Fixture JSON → ReplayProvider → Zod Parse → Fuzzy Matchers → Pass/Fail

The ReplayProvider reads the recorded fixture without network access. The Zod schema validates structural correctness, catching missing fields, type mismatches, and constraint violations. Fuzzy matchers then verify that free-text content has not drifted beyond acceptable thresholds. The CI pipeline aggregates all results into a pass/fail gate with a downloadable JSON report.

The complete harness consists of these files, each demonstrated above:

The recording layer handles fixture creation. eval/utils/recordFixture.ts captures live LLM responses with model version, temperature, and timestamp metadata, using atomic “wx” writes to prevent race conditions. Run it manually orils/recordEmbeddings.ts captures embedding vectors for reference and output texts; run it alongside recordFixture when prompt templates or reference texts change

The validation layer runs in CI without network access. eval/utils/ReplayProvider.ts reads fixture JSON and validates its shape with a Zod schema before returning typed responses, injected via vi.mock or constructor-based dependency injection. eval/schemas/product-description.schema.ts defines the Zod schema that acts as the contract between the LLM output and the consuming application. eval/matchers/toBeWithinEditDistance.ts provides a custom Vitest matcher for normalized Levenshtein distance, catching character-level drift in short text fields. eval/matchers/toBeSemanticallySimilar.ts provides a custom Vitest matcher for cosine similarity over pre-loaded embedding vectors, catching semantic drift in longer content.

The CI layer ties it together. vitest.config.eval.ts provides the dedicated Vitest configuration with eval-specific globs, timeouts, and reporter settings. .github/workflows/llm-eval.yml defines the GitHub Actions workflow for PR and nightly execution with fixture caching and artifact upload.

Best Practices and Pitfalls

Version fixtures alongside application code in the same repository. When a prompt template changes, re-record the affected fixtures and their corresponding embeddings to ensure the test baseline reflects the current prompt. Pin model identifiers (e.g., gpt-4o-2024-08-06 rather than gpt-4o) in fixture metadata so that you can detect provider-side model upgrades by comparing metadata, even before the eval suite catches output drift. Note that date-stamped model identifiers are an OpenAI convention; other providers may use different versioning schemes.

Keep similarity thresholds in a shared configuration file rather than hardcoding them in individual test files. This makes tuning simple when the team adjusts its tolerance for output variation across different features.

Don’ts

Never snapshot raw LLM strings as test assertions. They produce noise, not signal.

Do not call live LLM APIs during CI test runs unless the test is explicitly gated behind a LIVE_EVAL environment variable flag. Uncontrolled live calls introduce flakiness, cost, and rate-limiting failures.

Do not overlook the three Zod parsing modes: .strip() (the default, which silently removes unknown keys), .passthrough() (which preserves unknown keys in the parsed output), and .strict() (which rejects unknown keys with a validation error). Using .passthrough() allows the LLM to add unexpected fields without failing validation, which is useful during early development. Using .strict() rejects any fields not defined in the schema, which is appropriate for production contracts where unexpected output structure indicates a regression. The default .strip() mode silently drops unknown fields, which can mask regressions where the LLM introduces unexpected structure. Choose intentionally based on the maturity of the prompt and the criticality of the output schema.

Sharing our passion for building incredible internet things.