AI guides

Zero-HLM scales simulation demonstrations for whole-body humanoid manipulation

By 机器之心1 viewsSource: 机器之心
Zero-HLM simulation-to-real whole-body humanoid manipulation

The system reconstructs phone-captured scenes, generates and validates actions in simulation, then transfers them to G1. Tests cover three task types; expanded simulation data and real-data baselines differ in volume.

Article illustration

  • Paper: https://arxiv.org/abs/2610.35075
  • Project homepage: https://young-bit.github.io/zero-hlm/
  • Code: https://github.com/BIGAI-General-Vision-Lab/Zero-HLM

Last year, humanoid robots were still attracting attention by dancing and running. Now, people are beginning to expect them to enter factories and homes and take on concrete work. This requires robots not only to move, but also to manipulate objects while walking, coordinating their hands, legs, and bodies to complete tasks.

However, training generalizable whole-body mobile-manipulation models often requires large numbers of real-robot demonstrations. Changes to starting positions, object locations, or task configurations may require additional collection, creating continuing demands on people and equipment.

Does teaching humanoid robots to work necessarily require collecting so many real-robot demonstrations? Could they accumulate experience in simulation and then complete tasks in reality?

Real2Sim2Real offers a path: reconstruct a real scene as a simulation environment, generate demonstrations and train policies there, and transfer the skills back to a real robot. But two barriers remain before this path can support whole-body manipulation by humanoids.

The first barrier is turning a real scene into a usable training environment. Beyond appearance, dimensions and positions must be aligned, and collisions and joints configured. Humanoid robots walk and crouch, so scenes must also support changing viewpoints and contact relationships, making modeling and debugging more difficult.

The second barrier is efficiently generating executable whole-body demonstrations within the environment. Humanoid whole-body control must balance stability, contact, and motion tracking. Planned motions may therefore not execute accurately, and a small deviation in stance or hand trajectory can make a grasp miss, leaving data generation highly inefficient.

The problem thus becomes concrete: how can a real scene be turned into a reliable physical-interaction environment, and executable whole-body manipulation demonstrations generated, so that a humanoid trained for tasks only in simulation can complete them in reality and adapt to changes?

Recently, the Beijing Institute for General Artificial Intelligence, together with Beijing Institute of Technology, Tsinghua University, and Deta Intelligence, introduced Zero-HLM. Starting from scenes captured with a phone, the system uses Gaussian reconstruction as a reference to guide GPT-Astra in aligning scene geometry and configuring collisions and joints, creating a training environment that combines realistic appearance with physical interaction.

Within the reconstructed simulation environment, Zero-HLM generates whole-body task motions from source demonstrations, then uses execution feedback from the whole-body controller to jointly adjust the motions and scene, correcting actual execution deviations, expanding a small number of source demonstrations into successful, execution-validated trajectories under different configurations.

The method successfully expands 45 simulation source demonstrations into 4,800 executable demonstration trajectories. With task training using simulation data alone, the policy transfers zero-shot to the Unitree G1 and completes three types of long-horizon whole-body manipulation tasks. Across five generalization settings, the aggregate success rate reaches 69%, above the 23% of the real-demonstration training baseline.

The team also open-sourced Zero-HLM-Bench, containing three long-horizon tasks and nine skill stages, with a unified evaluation interface for different policies.

Article illustration

An overview of Zero-HLM's performance from simulation to reality.

01 / What challenges arise when teaching robots to work in simulation?

Teaching a robot to pick up an object from a table is already difficult. Having it first walk to the table, crouch to retrieve something, stand up and carry it, or even interact with objects using its feet adds another layer of complexity.

First, the world the robot sees keeps changing. Walking and crouching change the position and viewpoint of its head-mounted camera. Training scenes must look close to reality from different angles and also support physical interactions such as grasping, placing objects, and closing doors.

Second, planned motions do not necessarily execute successfully. A whole-body controller must handle motion tracking, body balance, and contact constraints together. When the robot approaches a target, a small positional or orientation error can cause the subsequent grasp to miss.

Simulation training therefore needs to solve two problems at once: scenes must look realistic and support reliable physical interactions; generated motions must also be validated through successful execution.

Article illustration

Figure 1 | The overall Zero-HLM framework: scene reconstruction, demonstration generation, execution validation, and real-robot deployment. Source: the paper's Figure 2.

02 / How does a phone-captured scene become a training environment?

Zero-HLM first performs Gaussian reconstruction using multiview footage captured with a phone, creating a digital version of the real room that can be viewed from different perspectives.

But "looking similar" is only the first step. To work inside it, a robot also needs collidable geometry, along with the moving joints and physical properties of objects such as cabinet doors.

Next, GPT-Astra uses the reconstruction as a reference to iteratively adjust the shapes and positions of furniture and objects, then configure collisions, joints, and other physical information. This aligns the visual and physical scenes. The table the robot sees must coincide with the tabletop it actually touches.

Article illustration

From phone capture to an interactive simulation scene: use Gaussian reconstruction as a visual reference to progressively construct and validate the physical scene.

03 / When execution deviates, data generation must adjust too

After building the scene, the team first teleoperates the robot in simulation, collecting 15 successful demonstrations per task, or 45 across three tasks. Starting from these source demonstrations, Zero-HLM adjusts interaction motions and replans movement for new starting points, object locations, and target positions, thereby generating new demonstration data.

Crucially, the generated motions must actually be executed in simulation by the same whole-body controller used at deployment. Checks must cover whether balance is maintained, whether any unreasonable collisions occur, and whether the task is completed.

If execution deviates, Zero-HLM uses execution feedback to jointly adjust the simulated scene configuration and reference motions.

For example, if the robot's final stance is off, it adjusts the positions of relevant objects to restore the required relative relationships; if the wrist height is unsuitable, it compensates the reference motion.

These adjustments happen during data generation and must still be followed by re-execution and validation. Only successfully executed trajectories enter the training set.

Article illustration

From source demonstrations to new trajectories: motion synthesis, physical-execution validation, and scene and motion adjustments based on execution feedback.

Finally, the system generates first-person images from the actual execution trajectories and pairs them with actions. Backgrounds use Gaussian rendering, while the robot and interacting objects use simulation rendering, so images reflect both realistic appearance and changes during execution.

In this way, 45 source demonstrations expand into 4,800 successful demonstrations. The entire expansion and validation process is automatic, requiring no additional human intervention or collection of new demonstrations for new configurations.

04 / How does training only on simulated task data perform on real robots?

The team uses the generated data to train the pretrained GR00T N1.7 model, then deploys it on G1 through the SONIC whole-body controller. The model predicts actions from first-person images, robot state, and task instructions, and the controller turns these into body movements.

Real-robot experiments include three task types: walking to a table and putting a banana in a basket; crouching to pick up a hat and moving it onto a cabinet; and carrying a fruit basket with both hands, closing a door with a foot, then placing the basket on a stool.

Article illustration

Figure 2 | Three real-robot task types and their base and extended configurations. Source: the paper's Figure 9.

Evaluation is divided into base and extended configurations. The former corresponds to the collection range of the real demonstrations; the latter changes the robot's starting point, grasping and placement locations, and object–target combinations to test whether the policy can adapt. These configuration changes are incorporated into training through simulation-data expansion, and model adaptability is validated on a real robot.

The 4,800 samples comprise 2,400 in base configurations, 800 for each of three tasks; 1,600 in extended configurations for Task A/B; and 800 with changed object–target combinations under Task A.

Besides the real-data baseline, the team sets up a GPT-6 Astra data-generation baseline: the large model generates and debugs simulation data-collection programs, and the resulting demonstrations train the same policy model.

Article illustration

Figure 3 | Task success rates on the real G1.

Real/Sim/Astra use 100/800/100 demonstrations per task, respectively; Astra has data only for tasks A and B.

The right plot aggregates the official website's four categories of settings, with Tabletop combining tests of grasping and placement positions. Source: the paper's Tables 8–9.

Under base configurations, with 100 demonstrations per task for both, the average success rates of policies trained on simulated and real data are 43.3% and 46.7%, respectively, showing similar performance.

When simulated demonstrations increase to 800 per task, the average success rate reaches 66.7%, exceeding the baseline trained on 100 real demonstrations by 20 percentage points. The three tasks reach 60%, 75%, and 65%, respectively. This shows the benefit of scaling simulated data; the two sides of the comparison do not use the same data volume.

The Astra-data-trained model succeeds at 20% on the banana task and 0% on the hat task; no usable successful demonstrations were generated for the basket-carrying task. This indicates that current large models still face challenges generating stably executable humanoid whole-body manipulation demonstrations, with a particular need to incorporate physical-execution feedback and validation of task success.

Under extended configurations, five settings involve 100 real-robot tests in total. The real-data baseline succeeds 23 times, while the model trained on expanded simulation data succeeds 69 times. The aggregate success rate rises from 23% to 69%.

After changing starting points, banana and hat task success rates reach 71.4% and 70%, respectively; after changing grasping and placement locations, they reach 67.5% and 50%. All 5 tests of changed object–target combinations succeed, compared with 1 success for the real-data baseline. Comparisons with the baseline in each setting appear in the figure above.

The value of simulation expansion lies in quickly incorporating new configurations into training, helping the model adapt to changes not covered by real demonstrations and reducing additional collection of real-robot demonstrations.

Article illustration

Model generalization with different training data: a real-robot comparison.

Article illustration

Zero-HLM's continuous generalization performance in real environments.

05 / From a single demonstration to repeatable training and evaluation

The team also introduces Zero-HLM-Bench, dividing three long-horizon tasks into nine skill stages. A unified interface allows comparison of different policies such as π₀.₅, GR00T N1.7, and DiT4DiT.

Evaluation examines not only whether the entire task succeeds, but also where the robot gets stuck. By combining different starting points and object positions, researchers can locate problems in movement, grasping, carrying, or placing, then target data expansion and model improvements accordingly.

Article illustration

Figure 4 | Skill-stage evaluation under different starting points and object positions. Source: the paper's Figure 12.

Unified environments, data interfaces, and evaluation protocols also make model comparisons and data-scaling studies easier to reproduce.

Under the unified evaluation protocol, GR00T N1.7 achieves a mean full-task success rate of 69.7%, compared with 64.0% for DiT4DiT and 59.0% for π₀.₅. DiT4DiT performs best on the hat-carrying task, reaching 80%.

Article illustration

Figure 5 | Full-task success rates of different models in Zero-HLM-Bench (simulation evaluation). Each model is tested 100 times on each task, and Mean is the arithmetic average of the three tasks. Data source: the paper's Table 13.

06 / Future research

Zero-HLM demonstrates a path for using Real2Sim2Real to develop generalizable whole-body humanoid manipulation skills: start from real scenes, generate demonstrations and train policies in simulation, then transfer to real robots, without collecting real-robot task demonstrations.

But the gap between simulation and reality remains. Diagnostic experiments show that simulated action supervision has value for transfer, while visual-domain differences remain an important bottleneck. Mixing real and simulated data does not automatically produce the best performance either. Aligning their visual distributions and coordinating training remain worth further exploration.

Validation currently focuses on three task types and their configuration changes. For homes and factories, reliability still needs to be tested with more objects, longer workflows, changing environments, and external disturbances.

This article comes from the WeChat official account "Synced," by "Synced."

Related topics

After reading

Join the AINRK community

Read how to join the community and connect with people working in AI.

Community information (opens in a new tab)