Know where a policy breaks before the robot does.
Evaluate across edge cases and real-world conditions, slice results by scene and embodiment, and trace every failure back to the frames and labels behind it.
Aggregate success rate hides the failures that matter in deployment. Agentuor's evaluation layer organizes held-out episodes by condition — lighting, clutter, occlusion, object novelty, embodiment — and reports performance per slice. Because evaluation shares a schema with annotation and collection, a failing slice links directly to the episodes, labels, and guideline decisions behind it, and agents propose which data to collect or relabel next.
Condition-based slicing
Performance by lighting, clutter, occlusion, object class, motion type, and environment.
Edge-case suites
Curated hard sets that grow automatically as reviewers flag difficult episodes.
Sim-to-real comparison
Track the gap between simulated and real performance on matched tasks.
Failure tracing
Jump from a failed evaluation episode to the source frames and their annotation history.
Data recommendations
Agents suggest collection or relabeling actions most likely to close a failing slice.
Versioned evaluations
Compare model versions against fixed suites with full reproducibility.
Step by step
Define suites
Assemble evaluation sets by condition, or let agents propose them from coverage gaps.
Run and score
Bring policy outputs or execution logs; results are scored against ground truth per slice.
Investigate
Drill into failures with linked frames, labels, and reviewer notes.
Act
Turn findings into collection and annotation tasks in the same workspace.
Common questions
Does Agentuor run the policy?
Agentuor scores outputs and execution logs you provide; it does not host or run policies itself.
Can evaluation sets be locked?
Yes. Suites are versioned and can be frozen for regression testing.
Which metrics are supported?
Task success, detection and segmentation metrics, trajectory error, and custom metrics via SDK.