Structuring robot episodes for learning
How to segment continuous recordings into episodes, attempts, and sub-tasks that models — and annotators — can actually use.
The episode is the unit of robot learning, yet most recordings arrive as continuous streams with no agreed structure. Getting the structure right early saves months of relabeling later.
Start from the task, not the recording
Define what a complete attempt at the task looks like — the state at which it begins and the states at which it ends in success, failure, or abort. Write these criteria down before anyone segments a frame. Ambiguity here becomes inconsistency everywhere downstream.
Separate episodes, attempts, and sub-tasks
- Episode: one complete engagement with the task, from initial state to terminal outcome.
- Attempt: a discrete try at a sub-goal within the episode, such as a grasp. Attempts can fail while the episode succeeds.
- Sub-task: a stage of the task with its own success criterion — approach, grasp, transport, place.
Label all three on a shared timeline. Models can then be trained and evaluated at whichever granularity the problem needs.
Anchor boundaries to state, not to appearance
A boundary placed where the video "looks like" a change drifts between annotators. A boundary placed where the gripper closes, the base stops, or the object leaves contact is reproducible. Where sensors permit, propose boundaries from state signals and have annotators confirm them.
Require an outcome on everything
Every episode and attempt should carry an explicit outcome. Missing outcomes are the most common defect in demonstration datasets, and the most damaging: a model cannot distinguish a demonstration from a failure it was never told about.
Capture metadata at ingest
Task, embodiment, scene, operator, and environment conditions are easy to capture at recording time and nearly impossible to reconstruct afterward. Treat them as required fields.
Put these ideas to work on your dataset.
Start with a walkthrough on your own data.