Durable execution for AI agents: how checkpointing survives crashes

Durable execution for AI agents means an agent's progress is saved as it runs, so when the process crashes or restarts it resumes from its last checkpoint instead of starting over. Because every agent step can cost real tokens and real time, resuming — rather than replaying from zero — is the difference between a production system and a demo.

Most agent code is written for the happy path. You wire up an LLM call, a tool, maybe a branch, and it works on your laptop. Then it runs on real infrastructure, a deploy rolls the process mid-run, a provider times out, or the box simply restarts — and the run is gone. Whatever the agent had done up to that point is lost, and if you retry, you pay for all of it again.

Durable execution is the property that makes that failure a non-event. This post walks through what it actually takes: checkpointing after every step, marking runs resumable on restart, not re-running work that already finished, and holding a run open at a human approval gate for as long as it needs — and why an agent frameworkalone doesn't give you any of it.

What durable execution actually means

Borrowed from workflow engines, durable execution means the state of a run is persisted continuously, so the run can always be reconstructed and continued from where it left off. Applied to AI agents, the unit of work is a node in a graph — an LLM call, a tool invocation, a condition — and the durability guarantee is: after each node completes, its result and the run's state are written to a database before the next node begins.

That single rule is what separates a system you can trust with production traffic from a script that happens to work. The checkpoint is the source of truth; the running process is disposable.

What happens when the backend crashes mid-run

Here is the sequence when a process dies partway through a run on a system built for durability:

• State is written to the database after every node, so the last checkpoint reflects exactly which nodes finished and what they produced.
• On restart, interrupted runs are flagged as resumable and continue from their last checkpoint — same state, same position in the graph.
• Parallel branches that finished before the crash keep their results; only unfinished nodes re-run. You never pay twice for completed work.
• Runs paused at an approval gate survive restarts indefinitely — a run can wait for a human for minutes or days without holding a process open.

Why re-running everything is the wrong answer

A naive "just retry the whole thing" approach is tempting because it's simple, but for agents it's expensive and unsafe. Expensive, because every re-run of a completed LLM node is another billed call. Unsafe, because tool calls have side effects — re-running a node that already sent an email, charged a card, or filed a ticket does it again.

Durable execution avoids both by resuming at node granularity: completed nodes are read from the checkpoint, and only the nodes that were actually in flight when the process died are re-executed. That's also why the order of operations matters — a durable system commits a node's result beforeadvancing, so a crash can never leave a step "half done" in the graph.

Durability is what makes human-in-the-loop possible

Human approval gates are the clearest case for durability. If a run has to pause and wait for a person to approve a step, you cannot hold a live process open for the hours or days that might take. The run has to be able to sleep as a checkpoint and wake up when the approval arrives.

That's exactly what a durable run does: a run paused at an approval gate is just a checkpoint waiting for an event. It survives restarts, deploys, and idle time, then resumes the instant a human approves — picking up at the next node with full state intact.

Why an agent framework alone doesn't give you this

Frameworks like LangGraph or CrewAI give you primitives — a way to define graphs, call models, and wire tools. What they don't give you out of the box is the production system around those primitives: checkpointed state that survives crashes, node-level resume, bounded parallel execution, approval gates that persist, per-step cost attribution, and an audit log. You can build those yourself, but that's the weekend (and the quarter) that turns into glue code.

This is the line between a framework and a runtime. A framework helps you describe an agent. A runtime runs it — durably, observably, and safely — so a crash, a deploy, or a slow human never corrupts a run.

How Ballast implements it

Ballast is a durable runtime for AI agents built on exactly these guarantees. State is written after every node; interrupted runs resume from their last checkpoint; finished parallel branches are preserved; and runs paused at approval gates survive indefinitely. Branch conditions run in a sandboxed expression interpreter, and every graph is validated before a single token is spent — so a malformed workflow fails at build time, not mid-run. Every step is cost-attributed and written to an audit log.

Where to go next

• See the primitives and the run model on the product page.
• Comparing options? Ballast vs Temporal covers durable execution built for general workloads versus built for agents.
• Ready to try it? Start free— the free tier includes checkpointed runs. You'll go from an empty terminal to a governed run in about ten minutes.