Community Article
Community articles are authored by SitePoint Premium contributors. Content is screened before publication, and SitePoint reserves the right to moderate or remove articles that violate our guidelines. Views expressed are those of the authors and do not necessarily reflect those of SitePoint.
Building an AI Flight Recorder for Healthcare: Monitoring Models, Data Access, and AI Decisions
SZShilei ZhangPublished inAI·Database·Databases·Data types·
August 19, 2026
The AI briefing for Developers
Stay up to date with AI tools, model releases, and developer workflows that matter.
Weekly. Free. One click to leave.
SitePoint Premium
Stay Relevant and Grow Your Career in Tech
- Premium Results
- Publish articles on SitePoint
- Daily curated jobs
- Learning Paths
- Discounts to dev tools
7 Day Free Trial. Cancel Anytime.
Introduction
Artificial intelligence is moving rapidly into healthcare.
Clinical documentation tools can summarize encounters. Predictive models can identify patients who may require additional care. Generative AI can assist employees with administrative tasks, search large knowledge bases, and help analyze complex healthcare data.
But as AI becomes embedded across healthcare systems, organizations face a new engineering problem:
How do you know what your AI systems are actually doing?
A healthcare organization may eventually operate dozens of AI-enabled applications. Some models may run internally. Others may be accessed through third-party APIs. An employee may use an approved enterprise assistant, while another application sends data to an external model provider.
Traditional application logs can tell us that a user logged into a system or called an endpoint. They often don’t provide a complete picture of the AI interaction itself.
For healthcare, that creates important questions.
Which AI model processed the request? What data did it access? Did the request contain sensitive information? Where was that information transmitted? Was anything retained? Which model version generated the response? Did another system subsequently use that output?
One approach is to build what I call an AI Flight Recorder: an independent observability layer that records the lifecycle of AI interactions without requiring developers to inspect the internal reasoning of the model.
In this article, we’ll design a simplified version using an AI gateway, structured event logging, policy enforcement, and an audit store.
The Black Box Problem
The term “black box” is often used to describe AI because it can be difficult to understand exactly how a complex model arrived at a particular result.
But healthcare organizations face another black-box problem.
They may not have sufficient visibility into the infrastructure surrounding the model.
Imagine a care-management application that sends information to an AI service:
Healthcare Application
|
v
AI Model
|
v
ResponseFrom an application perspective, this works.
From a governance perspective, however, several pieces of information are missing.
We don’t necessarily know which model version handled the request, what categories of information were transmitted, where the AI service processed the data, whether the request was stored, or which downstream application used the response.
Instead, I want the architecture to look more like this:
Healthcare Application
|
v
+----------------------+
| AI Gateway |
+----------------------+
| | |
v v v
Policy Audit Routing
Engine Logger Layer
|
v
AI Model
|
v
ResponseEvery AI interaction passes through a controlled layer before reaching the model.
The gateway becomes the observation point.
What Should the Flight Recorder Capture?
We don’t necessarily want to store every raw prompt and response. In healthcare, doing so could unnecessarily duplicate sensitive information and create another security liability.
Instead, the recorder can capture structured metadata about each interaction.
An event might look like this:
{
"event_id": "evt_8f291",
"timestamp": "2026-08-18T14:32:18Z",
"application": "care-management-assistant",
"user_role": "care_manager",
"model_provider": "approved-provider",
"model": "healthcare-assistant-v3",
"model_version": "3.2",
"purpose": "care_summary",
"data_classification": [
"PHI"
],
"storage_policy": "no-retention",
"destination_region": "us-east",
"latency_ms": 842,
"status": "success"
}Notice what isn’t included: the patient’s name, diagnosis, medical record number, or complete prompt.
The goal is observability without creating an unnecessary second repository of healthcare information.
Building the AI Gateway
A simplified Python service can sit between healthcare applications and approved AI services.
from fastapi import FastAPI
from pydantic import BaseModel
from datetime import datetime, timezone
import uuid
app = FastAPI()
class AIRequest(BaseModel):
application: str
user_role: str
purpose: str
prompt: str
@app.post("/ai")
def process_ai_request(request: AIRequest):
event_id = str(uuid.uuid4())
event = {
"event_id": event_id,
"timestamp": datetime.now(
timezone.utc
).isoformat(),
"application": request.application,
"user_role": request.user_role,
"purpose": request.purpose
}
return {
"event_id": event_id,
"status": "received"
}At this point, the gateway isn’t doing anything particularly intelligent. It simply establishes a centralized entry point.
Now we can begin adding governance.
Detecting Sensitive Data Before the AI Call
Before routing a request to an external AI service, the gateway can classify the information being transmitted.
For demonstration purposes, consider a simple detector:
import re
def detect_sensitive_data(text):
detected = []
if re.search(
r"bd{3}-d{2}-d{4}b",
text
):
detected.append("SSN_PATTERN")
if re.search(
r"bd{2}/d{2}/d{4}b",
text
):
detected.append("DATE_PATTERN")
return detectedA production healthcare system would require much more sophisticated classification. Pattern matching alone cannot reliably determine whether arbitrary text contains PHI.
However, the architectural principle remains useful:
Request
|
v
Data Classification
|
v
Policy Evaluation
|
+---- BLOCK
|
+---- REDACT
|
+---- ALLOW
|
v
AI ModelThe application no longer decides independently whether data can be sent to a model.
The policy layer does.
Creating an AI Policy Engine
Suppose an organization allows PHI only for explicitly approved AI services.
We can represent that rule in code:
APPROVED_PHI_MODELS = {
"healthcare-assistant-v3",
"clinical-model-v2"
}
def evaluate_policy(
model,
data_classification
):
contains_phi = (
"PHI" in data_classification
)
if (
contains_phi
and model not in APPROVED_PHI_MODELS
):
return {
"decision": "BLOCK",
"reason":
"PHI cannot be sent "
"to this model"
}
return {
"decision": "ALLOW"
}Now governance becomes executable rather than relying entirely on written policies.
The gateway can reject a prohibited request before data leaves the organization’s controlled environment.
Monitoring Where Data Goes
Model identity isn’t enough.
Healthcare organizations also need visibility into the destination.
Consider maintaining a model registry:
MODEL_REGISTRY = {
"healthcare-assistant-v3": {
"provider": "Provider-A",
"region": "us-east",
"retention": "none",
"phi_allowed": True
},
"general-assistant": {
"provider": "Provider-B",
"region": "unknown",
"retention": "unknown",
"phi_allowed": False
}
}def get_model_policy(model):
if model not in MODEL_REGISTRY:
raise ValueError(
"Unregistered AI model"
)
return MODEL_REGISTRY[model]This allows the flight recorder to associate every request with known infrastructure characteristics.
The resulting record could show:
Application:
Care Management
Model:
healthcare-assistant-v3
Provider:
Provider-A
Data:
PHI
Processing Region:
US-East
Retention:
None
Policy:
AllowedThat is far more useful to a security or compliance team than simply knowing that “AI was used.”
Creating the Audit Event
After policy evaluation, the gateway writes an event to the audit system.
def create_audit_event(
request,
model,
classification,
policy,
model_config
):
return {
"event_id":
str(uuid.uuid4()),
"timestamp":
datetime.now(
timezone.utc
).isoformat(),
"application":
request.application,
"user_role":
request.user_role,
"purpose":
request.purpose,
"model":
model,
"provider":
model_config["provider"],
"region":
model_config["region"],
"retention":
model_config["retention"],
"data_classification":
classification,
"policy_decision":
policy["decision"]
}These events can be stored in a centralized audit database.
The important principle is that the AI application shouldn’t be able to silently rewrite its own history.
For higher-assurance implementations, organizations could use append-only storage, cryptographic hashes, restricted write permissions, or other tamper-evident logging mechanisms.
Following the Entire AI Lifecycle
Logging the initial request is only part of the problem.
Suppose an AI system generates a recommendation that is subsequently displayed to a nurse or used by another application.
We should be able to trace that relationship.
Each interaction can therefore include a correlation ID:
correlation_id = str(uuid.uuid4())The same ID follows the workflow:
AI Request
correlation_id = ABC123
|
v
Model Response
correlation_id = ABC123
|
v
Provider Application
correlation_id = ABC123
|
v
Human Review
correlation_id = ABC123
|
v
Final Action
correlation_id = ABC123This creates something similar to distributed tracing in conventional software systems, but applied to AI workflows.
If an organization later investigates an incident, it can reconstruct the sequence of events.
Building an AI Governance Dashboard
Once structured events are collected, the data can feed an operational dashboard.
The dashboard shouldn’t attempt to display thousands of individual prompts. It should answer governance questions.
For example, an AI inventory view could show how many models are currently active, which applications use them, which models are approved for sensitive information, and where those services process or retain data.
A usage view could show requests by model, application, department, and purpose.
A data-governance view could highlight interactions involving PHI or PII, blocked requests, unknown destinations, and models with unclear retention policies.
A risk-monitoring view could surface unusual behavior such as a sudden increase in AI requests, attempts to use unregistered models, repeated policy violations, or applications unexpectedly sending sensitive data.
A Tableau, Power BI, or custom web dashboard could sit on top of the audit store:
AI Flight Recorder
+------------------------------------------------+
| Active Models | AI Requests | Blocked | Alerts |
+------------------------------------------------+
+----------------------+-------------------------+
| Usage by Application | Sensitive Data by Model |
+----------------------+-------------------------+
+----------------------+-------------------------+
| Processing Regions | Model / Version Changes |
+----------------------+-------------------------+
+------------------------------------------------+
| Recent Policy Violations |
+------------------------------------------------+The dashboard becomes the control center for understanding AI activity across the organization.
Why Existing Application Logs Aren’t Enough
Developers may reasonably ask why this requires another system.
After all, APIs already generate logs.
The difference is context.
A conventional API log might contain:
POST /generate
200 OK
842msAn AI governance event could tell us:
Application: Care Management
User Role: Care Manager
Purpose: Patient Summary
Data Classification: PHI
Model: Healthcare Assistant v3.2
Provider: Provider-A
Processing Region: US
Retention: None
Policy Decision: ALLOW
Latency: 842msBoth records describe the same request.
Only one explains the AI governance context.
The Architecture
A more complete implementation might look like this:
Healthcare Systems
EHR Claims CRM Portal
| | | |
+------------+-----------+------------+
|
v
+----------------+
| AI Gateway |
+----------------+
|
+--------------+--------------+
| | |
v v v
Data Classifier Policy Engine Model Registry
| | |
+--------------+--------------+
|
v
AI Router
|
+------------+------------+
| | |
v v v
Model A Model B Model C
| | |
+------------+------------+
|
v
Response Gateway
|
v
Healthcare System
Every stage
|
v
+-------------------------+
| AI Flight Recorder |
| |
| Request Metadata |
| Model / Version |
| Data Classification |
| Storage / Destination |
| Policy Decision |
| Response Metadata |
| Human / System Action |
+-------------------------+
|
v
Governance DashboardThis architecture separates three responsibilities: the applications that use AI, the infrastructure that controls AI access, and the recorder that provides independent visibility.
What the System Cannot Do
It’s equally important to define what an AI flight recorder cannot guarantee.
It cannot automatically explain the internal reasoning of every neural network. It cannot know what happens inside an external provider’s infrastructure beyond the technical and contractual visibility that provider exposes. And it cannot monitor employees who bypass controlled systems unless the organization has additional endpoint or network controls.
The flight recorder therefore isn’t a magical window into AI.
It’s an observability and control layer around AI.
That distinction makes the concept technically achievable.
Why Healthcare Needs This
Healthcare AI governance is often discussed as a policy problem.
Increasingly, it’s also an engineering problem.
Written policies may say which AI systems employees can use, but software needs to enforce those policies. Security teams need to know where sensitive information is traveling. Data teams need to understand which models depend on which datasets. Compliance teams need evidence showing how AI systems are being used.
As the number of models grows, manually maintaining spreadsheets of approved AI tools won’t scale.
Healthcare organizations may eventually need an AI control plane that can answer, almost immediately:
What models are operating?
Who is using them?
What data are they accessing?
Where is that data going?
Which model version produced this output?
What happened after the output was generated?
Those are fundamentally observability questions.
Conclusion
Healthcare doesn’t necessarily need another AI model.
It may need better infrastructure for controlling the models it already has.
An AI Flight Recorder provides one possible architecture: place a governance layer between healthcare applications and AI services, classify data before transmission, enforce model-specific policies, maintain a registry of approved AI systems, and create tamper-resistant records of important AI interactions.
The concept borrows from a familiar idea in as a reliable record of what happened when understanding the system matters most
As AI becomes part of clinical, operational, and administrative healthcare workflows, the ability to reconstruct how those systems interacted with people and data may become just as important as the models themselves.
For developers, that creates a new engineering challenge.
We aren’t only building AI applications anymore.
We also need to build the infrastructure that watches them.
