How to resume an AI agent after a crash

To resume an AI agent after a crash, its run state must be checkpointed to durable storage after every step. On restart, the interrupted run is detected as resumable and continues from its last checkpoint — same state, same position in the graph — re-running only the steps that were still in flight when the process died.

That is the whole answer, but it hides a lot of engineering. A crash is not a rare event in production: a deploy rolls the process mid-run, a provider hangs and the box is recycled, an out-of-memory kill lands on the wrong container, or the host simply reboots. If your agent has no durable memory of what it has already done, every one of those events throws away the run.

This post walks through what AI agent crash recovery actually requires — checkpointing after every step, detecting resumable runs on restart, re-running only the work that was interrupted, and surviving long human-approval pauses — and why an agent framework alone leaves you to build all of it.

The problem: a crash loses the whole run

Most agent code keeps its state in memory. The conversation so far, the outputs of the tools it has called, the branch it took three steps ago — all of it lives in process variables. That is fine until the process goes away. When it does, there is nothing on disk to reconstruct from, so the only recovery is to start the run again from the very first node.

Retrying from zero is worse than it sounds. Every re-run of a completed LLM node is another billed call, so a run that crashes near the end can cost you nearly double in tokens. It is slow, because you re-do minutes of work the user already waited through. And it is unsafe, because a node that already sent an email or charged a card will happily do it a second time on the replay. "Just run it again" is not a recovery strategy; it is a way to pay twice and hope nothing had side effects.

What "resume" actually requires

Resuming is only possible if the run left something behind to resume from. That something is a checkpoint: a durable record of the run's state and the results of the nodes that have finished. The single rule that makes it work is simple to state and unforgiving in practice — write the checkpoint to durable storage after every step, before the next step begins.

Once that rule holds, the relationship between the run and the process inverts. The checkpoint becomes the source of truth for where the run is; the running process is just a disposable worker that advances it. Any worker can pick up the run, because everything it needs to continue is in the checkpoint rather than in one machine's memory. Lose the worker and you have lost nothing that matters.

Detecting resumable runs on restart

Checkpointing is half the story; the other half is noticing there is something to resume. When a durable runtime starts up, it does not assume every run in the database is either finished or dead. It looks for runs that were mid-flight — a checkpoint exists, but the run never reached a terminal state — and flags each one as resumable.

A resumable run is then continued from its last checkpoint: same state, same position in the graph. Nothing about this depends on the run having crashed "cleanly." The process can vanish without any chance to run shutdown code, because the durability guarantee was met on the way in — the last write already happened, one step at a time, as the run progressed. Restart is just the moment the runtime reads that state back and keeps going.

Re-running only in-flight work

The point of a checkpoint is that you do not redo work it already contains. On resume, completed nodes are read straight from the checkpoint — their outputs are facts now, not something to recompute. Only the nodes that were actually in flight when the process died get re-executed. If ten nodes finished and one was running, you re-run one node, not eleven.

This granularity is what makes resume safe for side-effecting tool calls, not just cheap for token spend. The danger in any retry is doing an external action twice — sending the email again, filing the ticket again, charging the card again. A durable system avoids it by committing a node's result before advancing to the next node. Because the commit happens first, a completed action is recorded as completed, so resume reads it from the checkpoint instead of running it a second time. A crash can never leave a step half-done in the graph.

Parallel work follows the same rule. If a run fanned out into several branches and some of them finished before the crash, those branches keep their results. Only the branches that were still unfinished re-run. You never pay twice for a branch that already completed, and you never lose the ones that did.

Resuming runs that were paused for human approval

A crash is not the only reason a run has to survive without a live process. Human approval gates create the same requirement on purpose. If a run has to stop and wait for a person to approve a step, you cannot hold a process open for the minutes, hours, or days that might take — and a deploy or restart during that window must not discard the run.

Under the same durability model, this falls out for free. A run paused at an approval gate is simply a checkpoint waiting for an event. It is not occupying a worker; it is sitting in durable storage. It survives restarts and deploys indefinitely, then resumes the instant the approval arrives — picking up at the next node with full state intact. The same machinery that recovers a crashed run also lets a run sleep as long as a human needs it to.

Doing it yourself vs. a durable runtime

You can build all of this. Frameworks like LangGraph or CrewAI give you the primitives to try — a way to define a graph, call models, and wire tools together. What they do not hand you is the production system around those primitives: checkpointed state that survives crashes, node-level resume that re-runs only in-flight work, parallel branches that preserve finished results, and approval gates that persist across restarts.

That gap is the difference between a framework and a runtime. A framework helps you describe an agent. A runtime runs it durably — so that a crash, a deploy, or a slow human is a non-event rather than a lost run. Closing that gap yourself is the quarter of checkpoint plumbing, idempotency handling, and restart logic that quietly becomes the actual product you are maintaining.

How Ballast resumes runs

Ballast is a durable runtime for AI agents built on exactly these guarantees. State is written to the database after every node, so there is always a recent checkpoint to resume from. On restart, interrupted runs are flagged as resumable and continue from their last checkpoint — same state, same position in the graph. Parallel branches that finished before the crash keep their results, and only unfinished nodes re-run. Runs paused at an approval gate survive restarts indefinitely and resume when the approval lands.

You do not wire any of that together. The checkpointing, resumable-run detection, node-level replay, and durable approval pauses are the runtime's job — alongside bounded parallel execution, a visual builder, per-step cost attribution, and an audit log. What you write is the agent; what Ballast guarantees is that a crash never costs you more than the step that was in flight.

Where to go next

• See the run model and the durable runtime on the product page.
• Want the deeper mechanics of checkpointing? Durable execution for AI agents covers what it takes to make a crash a non-event.
• Ready to try it? Start free — the free tier includes checkpointed runs.