Community Article
Community articles are authored by SitePoint Premium contributors. Content is screened before publication, and SitePoint reserves the right to moderate or remove articles that violate our guidelines. Views expressed are those of the authors and do not necessarily reflect those of SitePoint.
Building a Human Review Pipeline for AI Search: A Practical Developer’s Guide
SASaifullah AdenwallaPublished inAI·APIs·Development Environment·Developer Tools·Browser Tools·
October 30, 2026
·Updated:October 30, 2026
The AI briefing for <a href="https://tooltechblog.com/how-applicant-tracking-systems-read-resumes-7-checks-developers-should-make/” title=”How Applicant Tracking Systems Read Resumes: 7 Checks Developers Should Make”>Developers
Stay up to date with AI tools, model releases, and developer workflows that matter.
Weekly. Free. One click to leave.
SitePoint Premium
Stay Relevant and Grow Your Career in Tech
- Premium Results
- Publish articles on SitePoint
- Daily curated jobs
- Learning Paths
- Discounts to dev tools
7 Day Free Trial. Cancel Anytime.
An AI search feature can return valid JSON, render correctly, and still give an unhelpful answer.
Imagine a documentation assistant answering a question about an API’s current authentication requirements. It retrieves an outdated page, summarizes it fluently, and displays a citation. Every technical component appears to work, but the user receives instructions that no longer apply.
Automated checks are valuable, yet they cannot replace a clearly defined judgment about whether an answer solves the user’s problem.
This guide builds a human-review workflow for a hypothetical API documentation assistant. You will define a review rubric, validate submissions with Python, measure reviewer disagreement, and turn accepted judgments into regression cases. The Python examples use Pydantic 2 and scikit-learn; they do not require a model-provider API key.
1. Separate Retrieval Quality from Answer Quality
Start by identifying the failure you want to detect.
A documentation assistant usually has several stages: receive a question, retrieve relevant documents, generate an answer, and present that answer with supporting material.
Do not collapse these into a single “good answer” rating. A low score without a reason tells the team little about what to fix.
SitePoint’sintroduction to retrieval-augmented generation provides background on how retrieval and generation fit together.
Preserve the exact question, answer revision, retrieved passages, document versions, and retrieval timestamp for each run. Store a snapshot manifest with stable passage IDs,supplied to that run, not today’s edited page. Keep the model and prompt configuration with the run so a later review can be traced to the behavior it evaluated
2. Write a Rubric Reviewers Can Apply Consistently
A useful rubric defines observable criteria and separates evidence support from task completion andd API version. Otherwise, reviewers may judge different questions while appearing to disagree about one answer
For an answer about API authentication, use separate labels such as:
Supported: every material factual claim is backed by the supplied evidence.
Partially supported: some material claims lack support, while others are grounded.
Unsupported: the answer’s central instruction lacks support or contradicts the evidence.
Cannot assess: the necessary evidence is missing, unreadable, or inaccessible.
Define task completion independently as complete, incomplete, or cannot assess. Forswer can faithfully summarize an obsolete page while failing the request for current instructions
Consider a fictional API migration: version 2 accepts X-API-Key, while version 3 requires Authorization: Bearer <token>. If the supplied evidence contains only the version 2 page, an answer repeating X-API-Key can be supported by its evidence but still use an outdatedinvents X-API-Key, label the answer unsupported. These are different engineering failures
Also establish what reviewers should do when theects its evidence while separately flagging that evidence as unsuitable for the user’s current task
Start with a calibration batch covering clear successes, unsupported claims, missing evidence, and version mismatches. Have two reviewers label the same items independently, then compare their reasons. Revise ambiguous definitions before increasing volume; the batch is for improving the rubric, not estimating production quality.
Keep a rubric version on every review. Labels produced under different definitions should not be mixed without checking comparability.
3. Make the Review Record a Validated Data Contract
A spreadsheet can help during exploration, but an engineering pipeline needs a stable record format.
Pydantic can validate a review record before it enters the dataset. Install Pydantic 2 and scikit-learn in your project environment, then use this model:
from pydantic import BaseModel, ConfigDict, Field, ValidationError
model_config = ConfigDict(extra=”forbid”, strict=True)
item_id: str = Field(min_length=1)
answer_revision: str = Field(min_length=1)
snapshot_id: str = Field(min_length=1)
reviewer_id: str = Field(min_length=1)
rubric_version: Literal[“auth-v1”]
“supported”, “partially_supported”,
“unsupported”, “cannot_assess”
completion: Literal[“complete”, “incomplete”, “cannot_assess”]
rationale: str = Field(min_length=20, max_length=2000)
Strict mode limits type coercion, and extra=”forbid” rejects unexpected fields. The model checks shape and allowed values; it cannot establish that a judgment is correct. A minimum rationale length does not guarantee a useful explanation. The snapshot ID references a manifest, which can also record that retrieval produced no usable passages.
On the server, resolve the item, answer revision, snapshot, and passage IDs against the stored run. Reject IDs that do not belong together. Obtain the reviewer identity from the authenticated session rather than trusting a submitted reviewer_id. Store a server-generated submission time and preserve review revisions instead of silently overwriting earlier judgments.
Run this after the model definition. It accepts one fictional review and rejects both an unknown label and an extra field:
“answer_revision”: “answer-02”,
“rationale”: “The answer invents X-API-Key; p1 requires Bearer.”,
review = Review.model_validate(payload)
{**payload, “support”: “mostly_fine”},
{**payload, “confidence”: 0.99}
Review.model_validate(invalid)
except ValidationError as error:
print(error.errors()[0][“type”])
raise AssertionError(“Invalid review was accepted”)
The expected error types are literal_error and extra_forbidden. Add separate integration tests for mismatched evidence IDs, unauthorized submissions, and retries. Use a database uniqueness rule or an idempotency key so a repeated request does not create a second independent review.
More reviews do not automatically produce better evidence. If a task is ambiguous, scaling it can simply reproduce the ambiguity.
The workflow needs clear eligibility requirements, examples, escalation routes, and acceptance criteria. Reviewers assessing API behavior may need domain knowledge that general writing fluency does not provide.
If you work with an external AI-data contributor service such as Acquirox, define the task contract before assigning work: required expertise, permittedp of the rubric, evaluation set, and acceptance decisions within the product team. Contributor capacity alone does not establish expertise in the API being reviewed
Pilot the actual task in the review interface. Track drafts, submitted reviews, items needing adjudication, and final decisions as distinct workflow states. Reviewer judgments should remain immutable once submitted; corrections should create a new revision with a reason. Keep adjudication separate from the original labels.
For data involving users or contributors, document permitted uses, access boundaries, and retention. Remove credentials and unnecessary personal information from review material. An internal support transcript should not become a broadly accessible annotation task by accident.
Distinguish annotation from usability research. A skilled reviewer can judge whether an answer is supported, but that does not prove a first-time user can understand the interface.
SitePoint’sguide to choosing usability tests and participants offers related guidance on matching participants and methods to the question being investigated.
5. Build a Review Interface That Reduces Guessing
The review interface shapes the data it collects.
Label Studio is an option for configurable labeling interfaces and data import/export. If you use it, map exported annotations into the Review schema through an adapter; do not assume its export structure matches your application contract. A custom interface may be appropriate when reviewers need API-version navigation or passage-level citations.
For the documentation assistant, show:
The original question and relevant user context.
The exact generated answer.
The evidence snapshot supplied to the model.
Separate controls for support and task completion.
A rationale field and an explicit “cannot assess” option.
Display stable passage IDs beside the evidence and let reviewers attach those IDs to their rationale. Show theed content as untrusted text or sanitized markup, and never treat instructions embedded in athird-party content inside the review panel
Avoid showing a preferred model name or a previous reviewer’s verdict before an independent judgment. For pairwise comparisons, randomize answer order and include appropriate tie or neither-acceptable outcomes.
Design for keyboard use, visible focus, readable text, and explicit selection states. Keep evidence and the answer accessible without requiring excessive horizontal scrolling.
Start all judgment controls unselected and validate them on submission. Use labeled radio groups, an accessible error summary, and visible saving/submission status. A failed save should preserve the draft. Disable duplicate submissions in the UI, but enforce duplicate handling on the server as well.
Practical check: observe a reviewer completing several borderline examples. Record where they pause, misinterpret a label, or cannot find the necessary
6. Use Disagreement to Improve the System
When two reviewers disagree, investigate before reducing their decisions to a majority vote.
An unclear definition.
Missing evidence.
Different assumptions about the user’s expertise.
An outdated source.
A genuine error by one or both reviewers.
Preserve independent labels and store the adjudicated result separately. This allows the team to study ambiguity instead of erasing it.
For two reviewers applying categorical labels to the same items, scikit-learn providesCohen’s kappa:
from sklearn.metrics import cohen_kappa_score
Keys identify the same item, answer, snapshot, and rubric.
(“q1”, “a1”, “s1”, “auth-v1”): “supported”,
(“q2”, “a1”, “s2”, “auth-v1”): “unsupported”,
(“q3”, “a1”, “s3”, “auth-v1”): “supported”,
(“q4”, “a1”, “s4”, “auth-v1”): “partially_supported”
(“q3”, “a1”, “s3”, “auth-v1”): “partially_supported”
raise ValueError(“Review sets must contain identical items”)
left = [a[key] for key in keys]
right = [b[key] for key in keys]
raw_agreement = sum(x == y for x, y in zip(left, right)) / len(keys)
print(f”Raw agreement: {raw_agreement:.2f}”)
print(f”Kappa: {cohen_kappa_score(left, right):.2f}”)
This toy example prints raw agreement of 0.75 and kappa of about 0.64. Four items are not enough to assess a review program. In production, check for duplicate reviews before building dictionaries, use one rubric version per comparison, report unmatched items, and align all records by item, answer revision, and evidence snapshot.
Kappa measures agreement beyond chance under its assumptions, not correctness. Inspect raw agreement, label frequencies, and the disagreement table alongside it. A single-category batch can make kappa undefined; report that limitation rather than substituting a reassuring score. Treat cannot_assess explicitly and report its frequency instead of silently excluding difficult cases.
If both reviewers apply the same flawed rubric consistently, agreement can be high while the evaluation remains unhelpful.
7. Turn Reviewed Examples into a Versioned Regression Set
Once difficult cases are adjudicated, turn them into regression cases with explicit acceptance criteria. For the fictional authentication task: use version 3 evidence, cite the passage requiring Bearer, and do not invent a header. If version 3 evidence is missing, an acceptable response should explain the gap rather than provide confident instructions.
Question and task category.
Expected behavior or acceptance criteria.
Rubric version.
Model and prompt configuration.
Retrieval configuration.
Review and adjudication records.
DVC can track larger datasets alongside Git. In an existing Git repository with DVC initialized and data/reviewed-auth.jsonl present, a minimal versioning step is:
dvc add data/reviewed-auth.jsonl
git add data/reviewed-auth.jsonl.dvc data/.gitignore
git commit -m “Version authentication review cases”
This commits metadata, not the dataset bytes. Configure an access-controlled DVC remote and push the data before expecting another machine or CI runner to retrieve it. Keep credentials and sensitive review material out of public repositories.
Keep development examples separate from held-out evaluation cases. If developers repeatedly tune prompts against the same questions, those questions become less useful as evidence of generalization.
Split related examples together where appropriate. Near-duplicate questions from one support conversation should not be scattered across development and evaluation sets.
Use two evaluation passes. First, hold the evidence snapshot fixed to compare answer-generation changes. Then test retrieval against the intended document versions to detect stale or irrelevant an overall improvement can hide a regression in version-specific or multilingual questions
SitePoint’sarticle on testing nondeterministic AI behavior provides further context for repeatable evaluation workflows.
8. Review Product Behavior Alongside Model Output
A correct answer can still fail in the interface.
Users may overlook a caveat, misunderstand whether an operation has been performed, or miss that the evidence applies to an older API version.
Pair answer review with practical usability tasks:
Find the authentication requirements for version 3 and identify the
Observe whether users complete the task and whether their confidence matches the available evidence.
SitePoint’soverview of user research methods helps distinguish task observation from preference surveys.
Turn interface failures into browser tests: a citation opens the expected document version, its passage ID matches the displayed evidence, missing evidence offers a recovery action, and a failed review submission preserves the draft. Verify that reviewers cannot access unassigned restricted items by changing an ID in the URL. These checks complement human judgments; they do not replace them.
Do not use a fluent response as a substitute for a clear interaction. The interface should distinguish retrieved information, generated interpretation, and actions the application has actually completed.
Start with a Small, Inspectable Pipeline
Start with one authentication task, a versioned rubric, and a small calibration batch. Use Pydantic for ingestion checks, Label Studio or a custom interface for review, scikit-learn for agreement analysis, and Git/DVC for dataset provenance. Add each tool only when it solves a concrete workflow need.
At release time, run deterministic checks for schema, evidence references, permissions, and interface behavior. Compare candidate answers against adjudicated acceptance criteria and send new ambiguous failures back for review. Record the dataset revision and configurations with the results so another developer can reproduce the comparison.
The purpose of human review is to produce evidence the team can act on. A smaller collection of well-defined, traceable judgments is often more useful than a large pile of ratings whose meaning nobody can explain.
