AI guides

EgoExplorer captures mobile manipulation with five trackers and a camera

By EgoExplorer团队0 viewsSource: 量子位
Wearable trackers for whole-body mobile manipulation

Shanghai AI Lab presents a wearable capture system using trackers, gloves and a camera, plus C³ motion reconstruction, and releases 100 hours of data and code. Reconstruction and multimodal generation are assessed separately; humanoid transfer is shown only in simulation.

Many real-world tasks in homes, laboratories and factories require manipulating objects while walking.

Walking toward a workbench, bending down to pick something up, turning to put it down—the grasp must remain secure throughout.

For humanoid robots to learn this kind of whole-body mobile manipulation from human demonstrations, the data must simultaneously record, on a common time base, vision, whole-body motion, finger movements and contact.

The Intern Robotics team at Shanghai Artificial Intelligence Laboratory has released EgoExplorer to address this need. It is a wearable vision-motion-touch data collection pipeline that requires no modifications to the environment, comprising a capture system and a whole-body motion reconstruction method.

The system consists of five trackers, data gloves and a head-mounted camera. It synchronously records first-person video, metric-scale whole-body motion, the joint poses of both hands, and palm tactile data, with expert-guided step-level annotations.

Article illustration
△ Synchronized playback of first-person video, whole-body reconstruction and hand tactile data

Article illustration

Why is data a bottleneck for mobile manipulation?

The central difficulty in mobile manipulation is the coupling of locomotion and manipulation. What a robot needs to reach, grasp or carry determines where it must move and the posture it must adopt. Meanwhile, its hands must maintain a stable grasp and fine control as it walks, turns or bends down. The two constrain each other at the same moment and cannot be separated into independent stages of 'walk first, then act.'

Robot teleoperation relies on specialized hardware and sustained human effort, making it difficult to scale. Human demonstrations offer a richer source of data. To use them for mobile manipulation, the data must meet two requirements at once.

First, the modalities must be complete. Vision provides task semantics and environmental context; whole-body motion describes walking, stance and support; finger joint poses describe how objects are grasped; and contact reflects force conditions that are difficult to judge from vision. The four modalities must be aligned to a common time base throughout continuous tasks to reconstruct how body movement and hand contact evolve together.

Second, collection must be scalable. Mobile manipulation takes place in varied real-world environments, including homes, laboratories and factories. Motion capture based on fixed camera arrays requires each recording space to be set up and calibrated beforehand, limiting data to a small number of locations. A capture system must be lightweight and easy enough to deploy that it can enter spaces already in use and be operated in parallel by multiple collectors in different locations.

Existing work has made important advances in its respective areas, but most records only part of this coordination. Representative examples include Ego4D, with 3,670 hours of video but no sensor-measured body or hand states; Nymeria, with 300 hours of wearable whole-body data but no finger joints or touch; and EgoTouch and OpenTouch, which provide tactile data over 20 and five hours respectively, but include no whole-body motion. Such partial recordings miss precisely the most essential part of mobile manipulation: the coupling between body movement and hand contact.

Article illustration
△ Comparison of capture modalities with representative prior work

In the comparison above, EgoExplorer's collection pipeline fully supports every dimension: metric-scale whole-body motion, nonvisual pose anchors, foot observations, finger joints, touch, mobile capture, mobile manipulation and expert-verified step-level annotations.

Five anchors, two gloves and one camera

Designing the capture system involves a trade-off between two approaches. Motion-capture studios provide dense measurements but cannot move, while video-only approaches are easy to deploy but can only infer occluded motion and contact. EgoExplorer takes a middle path: sparse tracking at key joints establishes the whole-body pose, while data gloves directly measure the fingers, with touch available as an option. All devices are wearable, commercially available off-the-shelf hardware:

Whole-body tracking: Five VIVE Ultimate Trackers are worn on the head, the backs of both hands and the outsides of both ankles. Their onboard cameras enable six-degree-of-freedom inside-out tracking without external base stations. The head and hand trackers provide metric-scale camera and end-effector trajectories; the ankle trackers record foot-placement positions and constrain stance and support outside the head-mounted camera's field of view;

First-person camera: A GoPro HERO13 is rigidly attached to the head tracker using a custom 3D-printed mount. The tracker directly measures the camera pose, eliminating the need to estimate it afterward using vision;

Hand capture: Two interchangeable glove types share the same tracker layout. MANUS Metagloves measure finger joint poses; Wuji gloves additionally record tactile signals covering 17 regions of the hand.

This compact, reusable setup takes only minutes to put on and supports continuous collection throughout the day. It requires no base stations, attached markers or other modifications to the environment. As long as the equipment remains worn, the collector's body-shape and mounting calibrations can be reused across tasks, so subsequent collection requires no recalibration. The entire setup is easy to transport and replicate, enabling simultaneous deployment at multiple sites and allowing the amount of data to scale with the number of collectors.

Article illustration
△ Capture apparatus schematic and examples from multiple environments

How can five tracking points reconstruct whole-body motion?

Recovering complete human motion from only five tracking points is underconstrained, with two main difficulties. First, each tracker's pose depends on two things simultaneously: the mounting offset between the device and the anatomical joint, and the pose of the body's kinematic chain itself. Different combinations of 'mounting offset plus joint pose' can explain the same observations. With direct fitting, mounting errors are absorbed into the estimated joint motion. Second, even when the anchors are accurately calibrated, intermediate joints such as elbows and knees remain undetermined. The reconstruction must also be physically plausible: feet must not penetrate the ground, for example, and body parts must not intersect one another.

To address this, the team proposed the C³ (Calibrate--Complete--Constrain) pipeline. Its central idea is to handle fixed body-sensor geometry separately from motion that varies over time, then perform completion and constraint in sequence:

Calibrate: The collector performs a prescribed sequence of movements, including standing, A/T poses, elbow bends, shallow squats and leg raises. The system automatically aligns the sequence with a reference motion library by movement stage and combines it with measured height to fit body shape, scale and the transformation from each sensor to its corresponding joint. This parameter profile needs to be estimated only once for each wearing configuration;

Complete: Calibration results convert tracker poses into anatomical anchors at the head, wrists and ankles. The team expanded AvatarPoser's input from 'head plus both hands' to all five anchors and trained it on AMASS as a temporal motion prior. It fills in the 17 joints not observed by the trackers and generates an initial whole-body trajectory, or 'motion proposal';

Constrain: The proposal is jointly refined at three levels. The skeletal level ensures consistency with measurements and temporal continuity; the surface level uses the subject's individual body shape to suppress intersections between body parts; and the ground level enforces consistency between the body and the ground, preventing feet from passing through it.

Finally, finger poses and tactile measurements from the gloves are fused into the reconstruction as direct evidence of interaction.

On the evaluation subset, using tracker-fitted results as the reference, the experiments produced the following results:

Whole-body point error was 18.0 mm, compared with 64.1 mm for the strongest baseline, AvatarPoser retrained with five anchors—a reduction of 71.9%;

For the 17 joints not covered by trackers, the error was 20.3 mm, comparable to the overall error. The advantage extended to areas away from the anchors, such as elbows and knees, indicating that the method recovers full-body joint motion rather than merely fitting the anchors;

Ablation experiments showed that each level contributes as constraints are added sequentially. Adding the skeletal level reduced body-point error from 79.0 mm for the motion proposal to 16.1 mm; adding the surface level then reduced intersecting pairs per frame from 72.8 to 38.7; adding the ground level further reduced the ground-violation rate from 61.4% to 6.0%. The full method combining all three levels achieved the best overall metrics. In addition, replacing the calibrated body shape and scale with an average body shape increased error from 18.0 mm to 59.5 mm. Omitting mounting calibration during training increased error to 95.0 mm; omitting it only at test time increased error to 98.1 mm. These results show that establishing body-sensor geometry before reconstructing motion is central to the method.

Article illustration
△ Comparison of the stages of hierarchical optimization

From laboratory to living room: what the pipeline collected

The team validated collection with this pipeline in five types of real-world environments: a chemistry laboratory, an industrial workshop, a study, a living room and a food-preparation area, spanning laboratory, factory and home settings.

The tasks focus mainly on professional activities and comprise ten task-template categories. Professional tasks include chemical laboratory procedures such as weighing, liquid-liquid extraction, reaction setup, rotary evaporation, oven drying and column chromatography. Everyday tasks include folding cardboard boxes, organizing bookshelves, folding clothes and making sandwiches. Each recording captures a continuous, complete task, preserving transitions between approaching a target, adjusting posture, manipulating objects and changing location. In addition to routine procedures, the collection retains 'recovery segments' showing how collectors adjust and continue after interruptions or failed attempts.

The pipeline's four data streams share a 30 Hz time grid: first-person video, SMPL-X whole-body motion, finger joints measured by the gloves, and a 24×31 tactile array for each hand.

Step-level annotation follows an expert-guided process. Conditioned on first-person video, task instructions and expert-supplied skill documents, a vision-language model segments each recording into subtasks. Each segment includes start and end times, a skill label, object and tool roles, and the observed outcome. The annotation workflow also includes human review: reviewers inspect each segment for temporal alignment and semantic consistency across video, reconstructed motion, touch and text, ensuring that descriptions correspond directly to synchronized observations.

Article illustration
△ First-person examples of the ten task categories

Recorded on humans, executed by robots: what can these data train?

The value of a collection pipeline ultimately depends on whether its data can support learning. Beyond validating reconstruction accuracy, the team conducted preliminary investigations into the training value of the collected data in two directions: multimodal world models and motion transfer from humans to humanoid robots.

Multimodal world models. The team modified Cosmos3-Edge to generate first-person video, motion derived from SMPL-X and touch simultaneously. In the experiments, each model received the same history of video, motion and touch. The comparison examined two settings: modeling each modality separately and generating them jointly. Joint generation outperformed separate modeling on every metric: video PSNR increased from 15.94 dB to 16.62 dB, contact F1 increased from 0.610 to 0.694, and motion MSE decreased from 0.059 to 0.044.

The team further examined the role of different conditioning modalities. With no additional conditioning, motion-generation MSE was 0.0478; conditioning on video recorded during the prediction interval reduced MSE to 0.0277; adding touch reduced it further to 0.0264. This indicates that video, motion and touch contain complementary information, and also underscores the value of synchronously collecting multiple modalities under the same clock.

From humans to humanoid robots. The team retargeted two recorded human motion sequences to a Unitree G1 humanoid robot, then used a ZeroWBC-style whole-body controller to track these reference motions in simulation. Human laboratory procedures involving weighing and reaction setup while moving were thus converted into whole-body humanoid motion. This provides a preliminary demonstration of motion transfer from human demonstrations to humanoid robots.

These studies remain preliminary: multimodal generation used a limited number of training steps, and humanoid motion tracking was performed only in simulation. The benefits of these data for learning on real humanoid robots still require further validation.

Article illustration
△ G1 humanoid reproducing a weighing procedure in simulation

Article illustration
△ G1 humanoid reproducing reaction setup in simulation

How to use it: open-source release plans

The reconstruction code has been open-sourced on GitHub. The full 100-hour dataset has been released on Hugging Face, with access requiring a Google Form, in LeRobot v3.0 format, allowing direct integration into existing robot-learning frameworks.

For research into humanoid whole-body mobile manipulation, hand-object interaction and tactile perception, EgoExplorer offers a pipeline for collecting human demonstrations at scale in real-world environments. The data can support areas including motion retargeting, imitation learning and world-model training.

Project page: https://internrobotics.github.io/egoexplorer.github.io/ Paper: https://huggingface.co/papers/2610.18872 Collection samples: https://huggingface.co/datasets/InternRobotics/EgoExplorer Code: https://github.com/InternRobotics/EgoExplorer

This article comes from QbitAI. Author: the EgoExplorer team.

After reading

Join the AINRK community

Read how to join the community and connect with people working in AI.

Community information (opens in a new tab)