Evaluating robot policies by condition, not by average
Why a single success rate hides deployment risk, and how to build evaluation suites that reveal where a policy actually breaks.
A policy with 92% success sounds ready. A policy with 99% success in good lighting and 60% under glare is not — and the average will not tell you which one you have.
Decide the conditions before you collect held-out data
List the conditions that vary in deployment: lighting, clutter, occlusion, object novelty, surface, embodiment, operator. Held-out sets should cover each deliberately, not by chance.
Build suites, then freeze them
An evaluation suite is a versioned set of episodes with fixed conditions. Once frozen, it becomes a regression test: every model version is scored on the same episodes, so improvements and regressions are real.
Report per slice, with counts
A slice with five episodes is an anecdote. Show the number of episodes behind every metric and treat thin slices as collection tasks rather than conclusions.
Grow edge-case suites from review
Episodes that reviewers flag as ambiguous or difficult are exactly the ones a policy is likely to fail. Route them into a dedicated hard suite automatically.
Trace failures to data
When a slice fails, the question is whether the training data covered that condition and whether it was labeled consistently. Evaluation linked to annotation history answers this in minutes; evaluation in a separate tool does not answer it at all.
Gate deployment on the worst slice that matters
Choose the conditions that carry safety or business risk and gate releases on those, not on the mean.
Put these ideas to work on your dataset.
Start with a walkthrough on your own data.