A humanoid locomotion policy we evaluated with a partner earlier this year reported ninety-two percent success across the held-out set. Sliced by surface, the picture was different: ninety-nine on tile, ninety-seven on carpet, eighty-eight on ramps, seventy-one on gravel, sixty-four on wet floor. The average was true and useless. The deployment environment had wet floors.
Why averages mislead for legged robots
Locomotion failures are driven by conditions the policy has under-learned, and those conditions are usually under-represented in the held-out set for the same reason they are under-represented in training: they were inconvenient to record. The average is dominated by the easy conditions, and the hard ones — the ones that cause falls near people — vanish into it.
Building terrain-aware suites
Condition-based evaluation starts before evaluation. Terrain, slope, surface wetness, payload, gait, and speed are recorded as metadata at collection. Held-out sets are assembled deliberately to cover each, with enough episodes per slice that a number means something. We treat any slice under thirty episodes as a collection task rather than a result.
Suites are then frozen. Every policy version is scored on identical episodes, so a change in the gravel number is a change in the policy, not in the test.
Labels that make slicing possible
Terrain slicing depends on labels that most locomotion datasets skip. Foot-contact events, proposed from force sensors and confirmed by annotators, let us score not just whether the robot fell but where in the gait cycle it began to fail. Terrain segmentation masks, fused from cameras and depth, let us attribute a slip to the surface actually underfoot rather than the surface listed for the session.
A fall is not a data point. It is a sequence of them, and the sequence tells you why.
From slice to fix
When the wet-floor slice failed, the agent-intelligence layer clustered the failures: most began with a heel-strike slip at speeds above a threshold. The recommendation was specific — collect forty wet-surface episodes at the two highest speed bands, add them to the hard suite, and check whether simulation randomizes friction low enough. The sim-to-real comparison showed it did not.
Gate on what matters
We now recommend that legged-robot programs gate deployment on the worst slice that carries safety risk, not on the mean. It is a harder bar. It is also the bar the people the robot will walk past would choose.