Evaluations

Checks turn “did the run finish?” into “did the run do the right thing?” A workflow carries a list of up to fifty checks. Each targets a state variable or a run metric and asserts something about it — the reply mentions the refund, the run stayed under five cents, the classifier output matches a known set. Scoring is automatic, additive, and completely detached from execution: it happens after a run has already succeeded, and a broken assertion becomes a failed result, not a failed run.

The seven check types

Checks fall into three families: text checks compare a state variable against a string, numeric checks compare a run metric against a number, and the expression check runs the same safe language as condition nodes.

The check object

A check is a small JSON object. Only name and type are always required; which of target and value matter depends on the type.

Validation rules

Checks are validated structurally before they are stored. Every problem found is reported at once, so a bad batch comes back with a list of reasons rather than the first failure. A workflow with any invalid check is rejected with 400 and none of the batch is saved.

{
  "detail": "Invalid checks: check 1: name is required; check 2: unknown type 'looks_good'; check 3: 'contains' needs a target state variable; check 4: invalid regex (missing ), unterminated subpattern at position 0); check 5: 'max_cost_usd' needs a numeric value; check 6: invalid expression (Function 'open' is not allowed)"
}

The evaluation context

When a run succeeds, the engine assembles a single context dictionary and evaluates every check against it. It contains:

A text check whose targetis not in the context does not error — it reads as an empty string, usually fails, and the detail is annotated (no 'target' in output) so you can tell a genuine miss from a typo. Inside an expression check, all of the above are available as names, plus the state alias — see the expression language.

Setting checks

{
  "checks": [
    { "name": "Mentions the refund", "type": "contains",
      "target": "draft_reply", "value": "refund" },
    { "name": "No internal error leaked", "type": "not_contains",
      "target": "draft_reply", "value": "Traceback" },
    { "name": "Category is known", "type": "regex",
      "target": "category", "value": "^(billing|refund|technical)$" },
    { "name": "Under budget", "type": "max_cost_usd", "value": 0.05 },
    { "name": "Under 8 seconds", "type": "max_duration_ms", "value": 8000 },
    { "name": "Reply is substantial", "type": "expression",
      "value": "len(draft_reply) > 40" }
  ]
}

{
  "id": "9bc62f209c52",
  "name": "Support Triage Crew",
  "version": 2,
  "checks": [
    { "id": "chk_a1b2c3d4e5", "name": "Mentions the refund", "type": "contains",
      "target": "draft_reply", "value": "refund", "case_sensitive": false },
    { "id": "chk_f6a7b8c9d0", "name": "Under budget", "type": "max_cost_usd",
      "target": "", "value": 0.05, "case_sensitive": false }
    // …the rest, each with a chk_ id
  ]
}

Reading results

A successful run whose workflow has checks carries an evalssummary. Runs with no checks — and every failed run — omit it entirely (evals is null).

{
  "id": "af772b3602ef",
  "status": "success",
  "cost_usd": 0.0035,
  "duration_ms": 4210,
  "evals": {
    "passed": 4,
    "total": 6,
    "results": [
      { "id": "chk_a1b2c3d4e5", "name": "Mentions the refund", "type": "contains",
        "passed": true,  "detail": "'draft_reply' contains 'refund'" },
      { "id": "chk_11223344ff", "name": "No internal error leaked", "type": "not_contains",
        "passed": true,  "detail": "'draft_reply' does not contain 'Traceback'" },
      { "id": "chk_55667788ee", "name": "Category is known", "type": "regex",
        "passed": true,  "detail": "'category' matches /^(billing|refund|technical)$/" },
      { "id": "chk_f6a7b8c9d0", "name": "Under budget", "type": "max_cost_usd",
        "passed": true,  "detail": "cost_usd 0.0035 <= 0.05" },
      { "id": "chk_99aabbccdd", "name": "Under 8 seconds", "type": "max_duration_ms",
        "passed": true,  "detail": "duration_ms 4210 <= 8000" },
      { "id": "chk_00ffeeddcc", "name": "Reply is substantial", "type": "expression",
        "passed": false, "detail": "expression did not hold" }
    ]
  }
}

The run_evaluatedevent is also pushed on the run's WebSocket stream with passed and total the moment scoring finishes — see Events & streaming.

Detail strings, per type

Each result's detail is generated by the check and states exactly what was compared. These are the formats:

Fleet-wide pass rate

Individual results roll up into a single quality signal across the whole workspace. The dashboard's metrics endpoint counts every check on every evaluated run.

{
  "total_runs": 412,
  "success_rate": 96.8,
  "total_cost_usd": 1.84213,
  "active_workflows": 5,
  "runs_by_day": [ { "date": "2026-07-07", "count": 34, "cost_usd": 0.1521 } ],
  "eval_pass_rate": 91.4,
  "evaluated_runs": 288
}

Setting checks from the SDK

Both SDKs resolve a workflow by id or exact name and replace its check list, returning the updated workflow. Results then arrive on each run as evals.

from agentos_sdk import AgentOS

client = AgentOS("http://localhost:8000", api_key="aos_live_…")

client.set_checks("Support Triage Crew", [
    {"name": "mentions refund", "type": "contains",
     "target": "draft_reply", "value": "refund"},
    {"name": "under budget", "type": "max_cost_usd", "value": 0.05},
    {"name": "no error leaked", "type": "expression",
     "value": "'error' not in draft_reply.lower()"},
])

run = client.run("Support Triage Crew", input={"ticket": "double charged"})
print(run["evals"]["passed"], "of", run["evals"]["total"], "checks passed")

import { AgentOS } from "@agentos/sdk";

const client = new AgentOS({ baseUrl: "http://localhost:8000", apiKey: "aos_live_…" });

await client.setChecks("Support Triage Crew", [
  { name: "mentions refund", type: "contains", target: "draft_reply", value: "refund" },
  { name: "under budget", type: "max_cost_usd", value: 0.05 },
  { name: "no error leaked", type: "expression",
    value: "'error' not in draft_reply.lower()" },
]);

One worked example per type

contains & not_contains

Both fold case unless case_sensitive is set. Given draft_reply = "We'll process your refund today.":

{ "name": "mentions refund", "type": "contains",
  "target": "draft_reply", "value": "refund" }
// → passed: true,  detail: "'draft_reply' contains 'refund'"

{ "name": "no stack trace", "type": "not_contains",
  "target": "draft_reply", "value": "Traceback" }
// → passed: true,  detail: "'draft_reply' does not contain 'Traceback'"

equals

Exact match after case folding. Given category = "Billing":

{ "name": "routed to billing", "type": "equals",
  "target": "category", "value": "billing" }
// → passed: true,  detail: "'category' equals 'billing'"   (case-insensitive)

{ "name": "routed to billing (strict)", "type": "equals",
  "target": "category", "value": "billing", "case_sensitive": true }
// → passed: false, detail: "'category' does not equal 'billing'"

regex

The value is a regular expression, searched against the target. Given order_id = "GB204815":

{ "name": "well-formed order id", "type": "regex",
  "target": "order_id", "value": "^[A-Z]{2}[0-9]{6}$" }
// → passed: true,  detail: "'order_id' matches /^[A-Z]{2}[0-9]{6}$/"

max_cost_usd & max_duration_ms

These ignore target and read the run metric. Given cost_usd = 0.0035 and duration_ms = 4210:

{ "name": "under 5 cents", "type": "max_cost_usd", "value": 0.05 }
// → passed: true,  detail: "cost_usd 0.0035 <= 0.05"

{ "name": "under 8 seconds", "type": "max_duration_ms", "value": 8000 }
// → passed: true,  detail: "duration_ms 4210 <= 8000"

expression

A boolean over the full context — state variables plus cost_usd, duration_ms, and input. It composes conditions no single typed check can:

{ "name": "substantial reply, on budget", "type": "expression",
  "value": "len(draft_reply) > 40 and cost_usd <= 0.05" }
// → passed: true,  detail: "expression held"

{ "name": "echoes requested locale", "type": "expression",
  "value": "input.get('locale') == 'en-US'" }
// → passed: false, detail: "expression did not hold"

// a check that raises never breaks the run — it just fails:
{ "name": "typo'd variable", "type": "expression", "value": "reploy != ''" }
// → passed: false, detail: "expression error: Unknown variable 'reploy'"

Blocking a deploy on a regression

Checks and experiments tell you a candidate got worse; an evaluation gate stops you from shipping it. Attach thresholds to a workflow and Ballast enforces them server-side at release time: when you cut a release, it evaluates the most recent completed evaluation for that workflow version against the gate and blocks the deploy with 422 if it regressed past your bar. An eval regression now preventsa production deploy — it isn't just a dashboard warning a human can miss.

$ curl -X POST ".../api/workflows/9bc62f209c52/release" \
    -d '{"environment": "production"}'

HTTP/1.1 422 Unprocessable Entity
{ "detail": "eval gate: check pass-rate 41.0% is below the required 90.0% — deployment blocked" }