Community Article
Community articles are authored by SitePoint Premium contributors. Content is screened before publication, and SitePoint reserves the right to moderate or remove articles that violate our guidelines. Views expressed are those of the authors and do not necessarily reflect those of SitePoint.
Designing a Production-Ready Agent Harness With Persistence and Checkpointing
Published inPython·DevOps·Cloud·Databases·
September 24, 2026
·Updated:September 25, 2026
The AI briefing for Developers
Stay up to date with AI tools, model releases, and developer workflows that matter.
Weekly. Free. One click to leave.
SitePoint Premium
Stay Relevant and Grow Your Career in Tech
- Premium Results
- Publish articles on SitePoint
- Daily curated jobs
- Learning Paths
- Discounts to dev tools
7 Day Free Trial. Cancel Anytime.
An agent run I was responsible for once got fourteen steps into a task and then vanished, because a routine deploy rescheduled the pod it was running on. No error in the logs worth the name. The orchestration process went away mid-run, took its in-memory state with it, and left a user watching a task that would never finish. The model had done nothing wrong. We just hadn’t built anything to catch a run when the ground moved under it.
That gap between “the model works” and “the system keeps working” is the agent harness, and it’s where I’ve spent a good part of the last two years, building the runtime that carries our agentic workflows across a pool of pods rather than a single process.
Before agents, I spent years on storage and distributed systems: replication that had to guarantee a write survived past the moment it was acknowledged, schedulers that handed work to nodes without doubling it up. Agent runtimes rhyme with that older work more than people expect. A run losing its progress to a pod eviction is a durability problem, and durability is well-trodden ground. What follows is the persistence and checkpointing layer that lets an agent survive interruption, with code you can run.
Everything here executes on the Python standard library. I’ve used SQLite so it runs with no setup, though the design carries over to Postgres, which is what we reach for when a run’s state has to outlive any single process.
What Production-Ready Actually Requires
Here’s the short list the harness has to get right. The rest of the article is an implementation of it.
- Treat every run as durable state in a store, not in-memory objects that die with process.
- Checkpoint at each meaningful transition, so recovery continues from the last good point.
- Make tool calls safe to run more than once, because after a crash you often can’t tell whether the last one excuted.
- Keep two workers from picking up the same run at once.
- Record enough at each step to debug failure nobody was watching.
I would have written a near-identical checklist for a job scheduler. The one difference here is that the work being coordinated is an LLM loop rather than a data block.
Modeling a Run as Durable State
Stop treating a run as a function call. Treat it as a record with an id, a position, and a history you can reload.
kind: str # “model” or “tool”
status: StepStatus = StepStatus . PENDING
steps: list = field(default_factory=list)
cursor: int = 0 # index of the next step to execute
status: str = “running” # running | completed | failed
version: int = 0 # incremented on every durable write
The cursor is what makes resumption work. On reload the harness reads it and knows which step is next, with nothing to replay. The version field keeps two processes from overwriting each other, and I’ll come back to it at the store. The full run serializes to one JSON document keyed by run_id. You could normalize the steps into their own table, and past a few thousand steps you might have to, but a single document reads back in one query on the recovery path, which is the path you most want to be fast and simple.
Deciding Where Checkpoints Belong
A checkpoint is a durable write of the run’s state. How often you take one is a cost trade, nothing more.
| Where you checkpoint | Cost while healthy | Work lost on a crash | When it fits |
| Once, at the end | Lowest | The whole run | Short runs cheap to redo |
| After every step | Moderate | At most one step | The default for most agents |
| Around each side effect | Highest | Nothing you can’t rebuild | Runs that move money or send messages |
A read-only research agent is fine checkpointing after each step. For anything that acts on the outside world I write twice around the acting step: once to say I’m about to do it, once to record what happened. The first write looks pointless right up until a pod dies mid-call. When the harness restarts and finds a step sitting in RUNNING, that saved status is the only reason it knows the call may already have fired.
A Persistence Layer That Survives Concurrency
The method that carries the weight is checkpoint. It does a compare-and-swap on version, so a write only lands if nobody has touched the row since we read it.
def checkpoint(self, run: Run) -> None:
“UPDATE runs SET data = ?, version = version + 1 “
“WHERE run_id = ? AND version = ?”,
(_serialize(run), run . run_id, run . version),
f”run {run . run_id} moved past version {run . version}”
The AND version = ? clause is the whole idea. If two workers both load version 7 and both try to write, one updates a row still at 7 and the other gets rowcount == 0 plus a StaleWriteError, which tells it it’s no longer the owner of this run and should back off. This is the same fencing pattern schedulers use to stop two workers from claiming the same job, a compare-and-swap on a version so only one writer wins. It’s old distributed-systems machinery pointed at a new problem. The payload is different now; the mechanism is the same.
The loop loads a run, finds the current step, and checkpoints on both sides of the work. Die at any line and a later call can load the same run_id and keep going.
step . status = StepStatus . RUNNING
store.checkpoint(run) # “about to execute this step”
step . response = model . call(run.goal, run.steps[:step.index])
step . response = tools . call(
idempotency_key=f”{run . run_id}:{step . index}”,
step . status = StepStatus . DONE
step . status = StepStatus . FAILED
store . checkpoint(run) # result (or failure) recorded
Follow a pod that dies right after the RUNNING checkpoint but before the result. On restart the harness loads the run, sees a step marked RUNNING, and can’t tell from the record whether the tool call went out. A model step is safe to rerun. A tool step is safe to rerun only if the tool was built for it, which is the next section.
The idempotency key is run_id plus step index. Because it’s deterministic, the retry after a crash generates the same key as the first attempt, and that’s what lets the tool layer spot the repeat.
I didn’t want to just assert that this works, so I ran the harness with a crash forced into a tool step and then resumed it. The real output:
=== attempt 1 (will crash at step 1) ===
[crash] pod died during step 1
state after crash: [(0, ‘model’, ‘done’), (1, ‘tool’, ‘running’)] | version: 4
final state: [(0, ‘model’, ‘done’), (1, ‘tool’, ‘done’),
(2, ‘model’, ‘done’), (3, ‘tool’, ‘done’)]
run status: completed | REAL tool side effects executed: 2
The run wanted two tool side effects and ran exactly two, even though step 1 was attempted in two separate processes. Recovery resumed from the durable running state instead of starting over.
Making Tool Calls Safe to Replay
The harness can’t promise a tool runs exactly once. What distributed systems give you is at-least-once delivery plus idempotency, and those two together get you the same result a caller would see from exactly-once. So the tool layer’s job is to make a repeat harmless.
def call(self, request: dict, idempotency_key: str) -> dict:
“SELECT result FROM tool_results WHERE key = ?”, (idempotency_key,)
return json . loads(row[0]) # key already ran; reuse its result
result = self . _invoke(request) # the real, side-effecting call
“INSERT INTO tool_results (key, result) VALUES (?, ?)”,
(idempotency_key, json . dumps(result)),
The in-process check is a floor. The stronger version sends the same key down to the external service when it accepts one; payment APIs are the familiar case, where an idempotency key means the provider itself won’t charge twice for it. When the downstream service dedupes on your key, you’re covered even in the nasty case where the first call succeeded remotely but your process died before recording it. Where a service offers nothing, I fall back to whatever natural key the operation already has, usually an upsert on (run_id, step_index), so a replay overwrites instead of duplicating.
Some side effects resist idempotency, and it’s better to say so than to pretend the pattern is free. Email is the one everyone runs into. The practical answer is a dedup table the mailer checks before sending, which shrinks the double-send window without fully closing it. For plenty of systems that bounded window is the honest target.
What I Watch Once It’s Running
Operating this surfaced a few things the design didn’t. Log the run_id, step index, and version on every transition. That version history is the nearest thing you get to a flight recorder, and I’ve untangled some genuinely odd incidents just by reading the run of version bumps for a single run_id.
The number I ended up caring about most wasn’t latency or token spend. It was how many steps came back as RUNNING at recovery time, since that’s a direct read on how often you’re dying mid-side-effect and relying on idempotency to cover it. When that count climbs, it’s usually an upstream dependency getting shaky under load, and this metric tends to move before the downstream error rate does, which makes it a useful early signal.
Put a TTL on RUNNING steps and alert past it. A step stuck in RUNNING well beyond any sane execution time is almost always a worker that grabbed the run and then died without letting go, and that alert also catches the leasing edge cases you can’t easily reproduce in a test.
Where to Take This Next
This is a single-worker harness with a durable spine. The version column is already doing lightweight fencing, so the obvious next move is real leasing: a worker claims a run by writing its own id and a lease expiry into the row, renews while it works, and other workers leave alone any run whose lease is still live. That turns the version guard into a coordination layer and lets a pool of workers pull from one shared queue.
The persistence model earns its keep away from recovery too. Once every run is a queryable record with its full step history, you can replay old runs against a new model and compare, which is a large part of an evaluation harness already. That path is worth following if reliability isn’t the only thing you want out of the state you’re already storing.
Of everything above, the cheapest idea is the one I’d hand you first. Write the checkpoint that says “about to execute” before you execute. It’s one extra round-trip, it looks redundant in review, and it’s the line that turns an ambiguous post-crash state into one you can recover from.
Nandish Nanjappa is a Staff Software Engineer at Attentive, where he keeps agentic AI workflows running reliably across a fleet of pods. He came to AI after more than a decade in storage and distributed systems, and he’s found most of the hard problems rhyme. He writes about the unglamorous infrastructure work that makes AI agents dependable.
