Long-horizon episodes, labeled consistently.
Mobile manipulation produces the messiest data in robotics: minutes-long episodes, many sub-tasks, partial successes, and objects that move, deform, or disappear. Agentuor keeps every boundary and outcome consistent across thousands of them.
The challenge
A single fetch-and-place episode can contain navigation, approach, grasp attempts, regrasps, transport, and placement — each with its own success criterion. When annotators segment these by hand, boundaries drift between people and across weeks, and outcome labels get applied to the wrong sub-task. Models trained on that data learn the inconsistency. Teams discover the problem only when a policy behaves erratically on a task the dataset supposedly covered.
Built for this data
Event segmentation assistance
Agents detect merged sub-tasks and propose boundaries at the frame where the robot's state actually changes.
Outcome enforcement
Every grasp attempt and every episode must carry a success, failure, or partial label before the batch can close.
Object tracks across views
Wrist, head, and external camera tracks reconciled into a single object identity with depth-consistent boxes.
Guideline drift detection
When two annotators place the same boundary differently, the disagreement is clustered and resolved once.
Per-sub-task evaluation
Score policies on grasp, transport, and placement separately so a failing stage is visible.
Demonstration filtering
Rank teleop demonstrations by cleanliness and route noisy ones out of imitation-learning sets.
Data and labels
- Synchronized RGB-D from wrist and head cameras with joint positions, gripper state, and force-torque
- Sub-task boundaries and per-attempt outcomes on a shared timeline
- 2D boxes, instance masks, and 3D cuboids for target and distractor objects
- Scene metadata: surface type, clutter level, lighting, object novelty
- Operator identity and intervention markers for teleoperated sessions
Typical workflow
Ingest sessions
Episodes are segmented from continuous recordings and tagged with task and scene metadata.
Pre-segment
Agents propose sub-task boundaries and grasp attempts for annotators to confirm.
Annotate and review
Object tracks and outcomes are labeled; ambiguous boundaries go to expert review.
Evaluate by stage
Policies are scored per sub-task and per condition, with failures linked to source episodes.
Outcomes teams work toward
Common questions
How long can episodes be?
There is no practical limit; the timeline tooling is built for multi-minute recordings with thousands of frames.
Can we define our own sub-task ontology?
Yes. Sub-tasks, outcomes, and attempt types are all configurable and versioned.
Do you support deformable objects?
Yes — mask and keypoint labels handle cloth, cables, and food items; agents flag identity ambiguity when shape changes.
Show us an episode. We'll show you the gaps.
Start with a walkthrough on your own data.