Evaluation & replay

Decide with evidence whether a new agent, workflow or policy version is good enough to ship — and see exactly what any run did.

Evaluation

The Evaluation page holds datasets, eval runs and release gates, plus shadow runs on live traffic and the dashboards that track it over time.

• Datasets — versioned sets of cases to test against.
• Runs — an eval run scores every case in a dataset version and produces a summary, per-case results and a report.
• Release gates — the datasets an agent must pass, the regression it may not exceed and whether human review is required. Check a gate evaluates the gates that apply to an agent spec.
• Shadow — run a candidate configuration alongside live traffic.
• Dashboards — results over time.

A gate that requires human review stays open until the review assignments are decided.

Observability

The Observability page — Everything an agent did, and why. — has four views:

• Timeline — search executions and decisions.
• Alerts — alert rules that fire on refusals and overruns.
• Evidence — export tamper-evident, signed evidence.
• Replay — replay a run.

Replay a run

Replay takes a run's recording — its steps, approvals and executions — and lets you inspect it, re-run it deterministically, or test a counterfactual to see what would have changed.