Reliability & checkpoints

What a checkpoint is

A checkpoint is a database row the engine writes at specific moments. It holds three fields that together are enough to reconstruct a run mid-flight:

{
  "state": {
    "input": { "ticket": "Charged twice" },
    "ticket": "Charged twice",
    "category": "billing",
    "draft_reply": "[mock:mock] ..."
  },
  "completed": ["classify", "draft"],
  "queue": ["approve"]
}

Checkpoints are written:

• after every node completes — including each node of a parallel wave, individually (step name after:<node_id>)
• when a run pauses at a human gate (gate:<node_id>)
• when a node fails and the run stops, with the failing node at the head of the queue (failed:<node_id>)
• when a gate is approved — the decision itself is checkpointed (approved:<node_id>)

The practical consequence: the blast radius of any crash is exactly one node. Everything up to the last checkpoint is durable.

What survives a crash and what re-runs

Resume always restores the last checkpoint and continues from its queue. That gives a simple, predictable contract:

Because agent and tool nodes commit their result to state only when they finish, an interrupted node leaves no half-written value — resume sees the clean state from before it started.

Failure mode 1: a node fails

A tool raised, an LLM call errored after its retries, or an expression referenced a missing variable. The step is marked failed with the error text; the run is marked failed with Node 'label' failed: reason; and a checkpoint is written with the failing node at the head of the queue. Resume re-runs just that node against the preserved state:

# curl — 400 if the run is not failed, or has no checkpoint
$ curl -s -X POST http://localhost:8000/api/runs/af772b3602ef/resume

# Python SDK
client.resume("af772b3602ef")

This is the right recovery for transient causes — a flaky API, a rate limit, a provider blip. Fix nothing, just resume; or fix the tool/config first and then resume. Resume is for failed runs only — a paused run is not failed, so resume returns 400 pointing you to /approve instead.

Failure mode 2: the process dies mid-run

On startup the backend sweeps every run stranded in running or pending and marks it failed with Interrupted by backend restart — resume from last checkpoint. Their last checkpoint is intact, so POST /resume picks up from the last completed node. Paused runs are deliberately left alone — they were already durable and simply resume waiting.

Failure mode 3: a node hangs — timeouts

Every execution attempt runs under a hard wall-clock timeout. A node that exceeds timeout_seconds is treated as a failed attempt — no agent or tool can wedge a workflow open forever. The timeout applies per attempt, so it composes with retries below.

Retries and exponential backoff

Both knobs live on every node type, alongside its type-specific config:

After a failed attempt the engine sleeps min(2^(n-1), 30) seconds before retry n, so the backoff sequence is 1s, 2s, 4s, 8s, 16s, 30s… (capped at 30s). With the default retry_count: 1 a node that keeps failing runs twice, one second apart, then surfaces the error.

{
  "id": "enrich",
  "type": "tool",
  "config": {
    "label": "Enrich Lead",
    "tool_name": "http_request",
    "params": { "method": "GET", "url": "https://api.example.com/enrich?email={email}" },
    "timeout_seconds": 30,
    "retry_count": 3
  }
}

Bounded loops

A loop node re-arms its body while an expression holds, but carries a hard max_iterations cap (default 5) that overrides the expression. The iteration counter lives in state under __loop_<node_id>and is checkpointed like everything else, so a loop's progress survives a crash too. A loop whose condition never turns false still exits at the cap — a runaway agent loop is a config error, not an outage.

Cancellation

POST /api/runs/{id}/cancel stops a running run mid-node — the in-flight node is abandoned and its work left uncommitted — or settles a paused one directly. Either way the run reads failed / Cancelled by user and keeps its checkpoints, so a run cancelled by mistake is still resumable. Cancelling an already-finished run returns 400.

$ curl -s -X POST http://localhost:8000/api/runs/af772b3602ef/cancel
# Python
client.cancel("af772b3602ef")

Idempotent triggers

Any trigger can carry an Idempotency-Key header. The first call with a given key starts a run and records the key; a repeat of the same key returns the original run instead of starting a duplicate. This is what makes retrying a trigger — from a webhook, a queue, or a crashed client — safe.

# curl — repeat this exact call and you get run af77... back, not a new run
$ curl -s -X POST http://localhost:8000/api/workflows/9bc62f209c52/runs \
    -H 'content-type: application/json' \
    -H 'Idempotency-Key: ticket-8841-triage' \
    -d '{"input": {"ticket": "..."}}'

# Python
client.run("Support triage", {"ticket": "..."}, idempotency_key="ticket-8841-triage")

Verify it yourself — the two-minute durability test

The single most convincing check: pause a run, kill the backend outright, restart it, and finish the run. It works because the pause checkpoint is already in the database.

# 1. run a gated workflow so it pauses
$ curl -s -X POST http://localhost:8000/api/workflows/9bc62f209c52/runs \
    -H 'content-type: application/json' -d '{"input": {"ticket": "test"}}'
# -> { "id": "af772b3602ef", "status": "pending" }   # becomes paused within a moment

# 2. kill the backend, hard
$ pkill -9 -f "uvicorn main:app"

# 3. start it again — startup sweeps interrupted runs; the paused one is untouched
$ uvicorn main:app --port 8000 &

# 4. the run is still paused; approve it
$ curl -s -X POST http://localhost:8000/api/runs/af772b3602ef/approve \
    -H 'content-type: application/json' -d '{"approved": true}'
# -> status: success