AI guides

Zhejiang University and Unitree open-source Movens-5M for multimodal whole-body motion pretraining

By 机器之心0 viewsSource: 机器之心
Movens-5M data and Movens-Zero whole-body robot motion

Movens-5M pairs five million simulation-validated robot motions with text, video, speech and music. The team trains Movens-Zero and demonstrates it on a Unitree G1, using scaling experiments to examine gains from more data and larger motion experts.

The Zhejiang University–Unitree Joint Laboratory for Embodied Intelligence and other teams have built and open-sourced the pretraining dataset Movens-5M from a vast collection of internet videos of humans, containing five million robot motions validated through simulated execution, totaling more than ten thousand hours.

The dataset covers 297 specific actions, including walking, squatting, mopping and boxing, across sixteen categories of settings such as home life, fitness training and outdoor recreation.

The team pairs text instructions, video demonstrations, speech and music with robot motions to achieve multimodal alignment, and provides motion descriptions by time segment and body part. Researchers can train models on these data to learn which motions correspond to different forms of input.

The motion model trained on the dataset, Movens-Zero, can generate whole-body robot motions from all four input types. The team demonstrated execution on a real Unitree G1 and used scaling experiments to test whether more data and a larger motion-generation module could yield continued improvements.

Article illustration

Overview of Movens-Zero. Images: paper front page and Figure 1.

How does a video of someone wiping a door become robot-training data?

What must a model-training example contain beyond video? Consider a cleaning clip first.

In the clip, a person bends down to wipe the lower-middle part of a door, then gradually stands upright and raises an arm. This data record uses four views to show the original video, 3D human motion reconstructed from it, motion converted for the G1, and the G1's execution in simulation.

Article illustration

A cleaning-data example. From left to right: Original Video, Recovered Motion, G1 Reference and G1 Tracking in simulation.

The system first estimates the person's 3D motion from video, then adjusts it to the robot's body structure. It then has a simulated G1 execute the adjusted motion to check whether the robot can complete it stably.

The record also includes text describing each body part's movements over different intervals. This 7.25-second video can be divided into three stages:

0–4 seconds: The person bends forward, holding a green cleaning cloth in the left hand and wiping the lower-middle part of the door back and forth. The right arm is bent and positioned behind the body.

4–6 seconds: The person gradually straightens the upper body, raising the left arm as the wiping position moves upward.

6–7.25 seconds: Both arms rise, reaching toward a higher part of the door.

These descriptions are like step-by-step video instructions, recording when the person bends, which hand wipes, and when the person stands and raises an arm. Each step corresponds to video frames and robot motions, helping the model learn to generate motion from text requirements.

The data also contain object-interaction motions, including dumbbell training and barbell deadlifts. Dumbbell training includes curls and bending forward; deadlifts involve leaning the upper body forward, slowly lowering the barbell and standing upright again. These scenes show how arms, legs and torso coordinate their movements.

Article illustration

Object-interaction examples: recovered human motions and G1 animations for dumbbell training and barbell deadlifts. The G1 barbell scene is a reference-motion playback.

How do millions of videos become five million robot motions?

The cleaning and object-interaction examples show how one record connects video, text descriptions and robot motions. Scaling such samples to five million requires an automated pipeline capable of processing vast video collections, identifying human motion and converting it into trajectories robots can execute stably.

Occluded people and camera shake can interfere with motion estimation. Human and robot body proportions differ, as do their joint ranges, and robots must also maintain balance while moving. The system must therefore adjust motion to the robot's body structure before validating stable execution.

The team built an automated data-processing pipeline with five stages: clip generation, quality filtering, semantic annotation, motion recovery and physical refinement in sequence.

Article illustration

Movens-5M data-processing pipeline. Image: Figure 2 of the paper.

  • Clip Generation: The pipeline first splits long videos at shot boundaries into continuous clips, preserving source information.
  • Quality Filtering: The system checks image quality, whether the person is clearly visible and whether sufficient motion information is present, removing unsuitable material.
  • Semantic Annotation: The system adds scene and action descriptions, describes changes in each body part over time and organizes corresponding text and available speech and music information.
  • Motion Recovery: The system estimates 3D human poses from video frames, recording how the body moves over time.
  • Physical Refinement: The system checks and fixes jitter, instability and other motion problems, adjusts motion to the G1's body structure and finally tests stable robot execution in simulation.

Through this pipeline, the team built a collection containing fifteen million human motions, five million robot motions, bringing together more than ten thousand hours of behavior data. The text also records how each motion changes over time and how the different body parts move.

These paired data let the model learn which actions a robot should take in response to a text or spoken instruction, a piece of music or a video demonstration.

One model responds to every input modality using these data

The team trained Movens-Zero on the paired data, teaching one model which actions correspond to text, speech, music and video. After training, it could generate whole-body robot motions from all four inputs, with real-world execution demonstrated on a Unitree G1.

You can tell it what to do in text. In the demonstration, entering "Dance some hip-hop" generated the corresponding motion without requiring an additional reference video.

Article illustration

Text input: Dance some hip-hop.

You can also speak the instruction directly. In the demonstration, the robot performed the corresponding action after hearing "Salute with your left hand."

Article illustration

Speech input: Salute with your left hand.

Music can drive whole-body motion. The model generates steps and upper-body movements according to the music's rhythm and style.

Article illustration

Music-driven motion demonstration. The animation is silent; see the project website for a video with audio.

A video demonstration can also be used directly. The input may be an internet clip or footage captured on site. Based on the person's demonstration, the model generates motions suitable for robot execution.

Article illustration

Real-robot demonstration with video input. The inset on the left shows the reference video; the right shows the Unitree G1.

Movens-Zero first analyzes text, sound or video through an understanding module. The same motion-generation module then combines the input with the robot's current state to plan subsequent whole-body motion. The paper calls this module the motion expert. The robot controller subsequently sends commands to individual joints to execute the generated motion.

Article illustration

Movens-Zero's understanding, generation and execution architecture. Image: Figure 7 of the paper.

During live interaction, people may change their spoken requirements or demonstrate new actions. The model must update the robot's subsequent motion accordingly.

Movens-Zero generates a short motion segment each time, and the robot executes its initial portion first. The model keeps the unexecuted tail as the beginning of the next segment, then generates subsequent motion using the latest input and robot state, smoothly connecting successive segments. In this way, Movens-Zero connects understanding, generation and execution into a real-time closed loop.

Article illustration

Live video input: the model continuously generates subsequent motion from the latest frames and the robot's body state.

Can gains continue with more data and larger motion experts?

The preceding demonstrations show what the model can do. Scaling experiments further test whether increasing training data and enlarging the motion expert make generated movements better match input requirements and improve robot execution. The team examined the effects of data and motion-expert size separately, then compared the benefits of scaling both.

Article illustration

Data and model scaling, Figure 9 of the paper. The six upper curves show motion-generation learning error, text matching, video-motion error, music-beat alignment, speech matching and motion tracking as scale changes. The lower charts show gains for different module sizes as data increases from 10% to 100%. ↑ means higher is better; ↓ means lower is better. Enlarge the image for details.

In the chart, moving right along the horizontal axis means more parameters and a larger motion-generation module; different colors indicate different proportions of training data. M on the axis means millions of parameters: 64.2M, for example, is 64.2 million parameters.

Following one curve from left to right compares performance as the motion-generation module grows. Comparing differently colored curves at the same horizontal position shows the effect of more data on a module of the same size.

More data improves both generation and tracking

In the first experiment, the team fixed the motion expert at the 910.5M-parameter XL size and increased training data through these proportions of Movens-5M: 1%, 3%, 10%, 30%, 100%. Each level included the preceding level's data. Total deduplicated motion duration rose from 122 to 12,231 hours, with more motion examples available as well.

As the training-data proportion rose from 1% to 100%, the score for matching text requirements to generated motion increased from 12.72% to 62.57%. The paper denotes this metric R@3. A higher score means better alignment between text and motion.

For video input, motion error measured by MPJPE fell from 132.82 to 40.90 millimeters, or from approximately 13.3 to 4.1 centimeters, a reduction of 69.2%. This metric measures the mean difference in body-joint positions between generated and reference motions. Lower is better.

Speech-instruction matching also rose from 13.98% to 74.30%, while music BeatAlign increased from 0.23 to 0.61. Tracking Score rose from 0.21 to 0.72, indicating improved overall tracking of reference motions.

Further gains persisted when data expanded from 30% to 100%: text–motion matching increased another 20.77 percentage points, and video-motion error fell a further 53.5%.

Larger motion experts further improve generation and tracking

In the second experiment, the team used all training data to train four motion-expert sizes, S, M, L and XL, with parameter counts of 64.2M, 187.9M, 500.1M, 910.5M, increasing from 64.2 million to approximately 910 million. The model responsible for understanding text, sound and video stayed fixed; only the motion-generation module's size changed.

Scaling the motion expert from S to XL reduced validation flow-matching loss, Val. FM Loss, from 0.487 to 0.159, a decrease of 67.4%. This metric measures learning performance in motion generation; lower is better. Tracking Score simultaneously rose from 0.18 to 0.72, while relevant metrics for text, speech, music and video inputs also improved continuously.

Larger motion experts benefit more from additional data

The bar charts below Figure 9 show another comparison: how much do different motion-expert sizes improve when given the same increase in training data?

Take text–motion matching R@3. Increasing data from 10% to 100% raised the S expert's score from 6.34% to 10.30%, a relative improvement of 62.5%; the XL expert's score rose from 24.92% to 62.57%, a relative improvement of 151.1%. The other five metrics show the same pattern: among the four sizes compared, greater capacity produces larger relative improvements from added data.

These experiments show that more data and larger motion experts can work together to further improve motion generation and execution. More training data teaches a richer range of motions, while larger motion experts can make fuller use of those data.

Accessing the data, models and paper

Researchers can use the open data and models for further motion-generation and scaling experiments. The project website provides access to Movens-5M, trained models, code and other resources. The team also supplies Movens-Intent, a companion test set for evaluating motion generation from text, speech, music and video. The real-robot demonstrations and data examples come from the project website; the method diagrams come from the paper.

  • Dataset: huggingface.co/datasets/wendell0218/Movens-5M
  • Project website: movens-zero.github.io
  • Paper: arxiv.org/abs/2610.41877

This article was originally published by Synced. Author: Synced.

Related topics

After reading

Join the AINRK community

Read how to join the community and connect with people working in AI.

Community information (opens in a new tab)