RoboScience’s WorldFlow jointly predicts future video and objects’ 3D trajectories

WorldFlow jointly predicts video and 3D motion, trained on about 1.44 million multisource samples. In the team’s 12-task physical-robot test, manipulation success rose from 11.1% to 66.7% using the same grasping and control system as μ₀; the result applies to that tested setup.
Editorial verification: The methods and figures in this article come from the WorldFlow team’s paper and project page, and are not an independent reproduction. “World’s first” was not supported by a sufficiently verifiable technical definition or scope of prior work, so it is not retained as an established conclusion; related wording in original figures and their historical descriptions should be read as a source claim. The physical-robot results are a small-sample comparison across 12 selected tasks, with 36 attempts per method: trajectory success was 35/36 versus 9/36, while end-to-end execution was 24/36 versus 4/36. These are different measures. ADE/FDE reductions apply to the listed benchmarks and ablation conditions and do not establish equivalent gains across all physical scenarios or robot embodiments. Original WorldFlow paper (opens in a new tab)Team experiments, denominators and future work (opens in a new tab)
RoboScience recently released WorldFlow, a world model that jointly predicts video and 3D trajectories. It enables a robot to understand manipulation intent from its current scene and task instruction, imagine how objects should move next in three-dimensional space, and thereby make more precise object-motion predictions for robotic manipulation.

For a human reaching for a cup, the brain has already simulated grasping, moving and placing it. Such instantaneous anticipation of the physical world underpins all manipulation. Yet robots still face two major challenges in truly imagining the physical world:
- Video-generation models accumulate rich experience of physical interactions from vast amounts of video and can generate realistic future frames. But an object’s 3D motion is implicit in changes to 2D pixels. Extracting that physical information still requires methods such as geometric reconstruction and motion tracking, inevitably introducing accumulated error.
- Methods whose sole generation target is 3D motion may not fully use the appearance, contact and occlusion cues in video for training supervision, although they can still accept visual conditions such as images. In its chosen ablations, the team found that joint video-and-motion generation reduced trajectory-prediction error. This does not justify claiming that all direct-prediction methods lack visual signals or cannot generalize.
This prompted the team to reconsider: Could the rich visual information in video let a model directly understand and predict three-dimensional motion?
Following this idea, the team released the WorldFlow world model to learn to predict motion in the 3D physical world from real and simulated scenes. It connects visual representations with 3D motion, providing more accurate 3D dynamics predictions for robotic manipulation.

- Paper: WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining
- Link: https://arxiv.org/abs/2610.39278
- Project page: https://worldflow-rbs.github.io/
Native 3D dynamics prediction:
Exploring robots’ prediction of future three-dimensional motion
WorldFlow’s central innovation is treating 3D motion as an independent generation modality alongside video. Instead of extracting trajectories after generating video, the model simultaneously predicts future frames and objects’ 3D motion in a single generation process.
Given an observed scene, a language instruction and a target object, WorldFlow jointly predicts the future video and the target object’s 3D trajectory. In one example from the paper, the instruction is “Pick up the black slipper and place it on the third shelf of the shoe rack.” WorldFlow can preview the slipper being picked up, moved and placed on the rack, while directly predicting the paths and final positions of points on the slipper in 3D space.

To bridge the inherent gap between 2D images and 3D motion and tell the model which moving 3D point corresponds to which pixel, the team designed two complementary encoding mechanisms: 3D point geometric encoding helps the model understand objects’ positions and geometric relationships in three-dimensional space; spatiotemporal positional encoding aligns motion trajectories with video frames, allowing the model to connect a particular object’s motion with the relevant visual regions.
In the Mixture of Transformer (MoT) architecture, video and motion exchange information through bidirectional attention and are jointly denoised in a unified generation process. Video provides visual cues such as object appearance, contact and occlusion, while 3D motion provides spatial-displacement information.
Below are the model’s inference results in real home environments:






Pretraining on over a million samples from multiple sources:
Learning patterns of physical motion from large-scale data
To teach WorldFlow diverse patterns of object motion, the team built a data pipeline integrating 17 data sources from simulated scenes, robot-manipulation scenes and human-manipulation scenes, assembling approximately 1.44 million training samples. Each sample contains both a manipulation video following a task instruction and the manipulated object’s 3D motion trajectory. The model thus learns not only “how the image should change” but also “how the object should move,” acquiring 3D motion priors from diverse tasks and scenes.
To evaluate dynamics prediction in robotic settings, the team also built WorldFlowBench, covering simulated, real and spatial-cognition scenarios. It measures trajectory-prediction accuracy through Average Displacement Error (ADE) and Final Displacement Error (FDE).
Ablation evidence:
Why does joint generation perform better?
Adding video supervision outperforms motion-only prediction
In ablation experiments, the team compared WorldFlow with a model variant that predicts only 3D motion. Both used the same network architecture, training data and hyperparameters; the difference was whether video generation was included. Adding the video-supervision signal substantially improved 3D motion prediction: ADE fell by 48.5%, 19.8% and 55.9% in simulated, real and spatial-cognition scenarios, respectively, while FDE fell by 54.4%, 25.7%, 68.5%.

Directly generating video and motion trajectories outperforms “generate video first, then track trajectories”
The team compared WorldFlow with two-stage approaches that first generate video and then track 3D trajectories using methods such as Track4World or 4RC. Compared with Track4World, WorldFlow reduced ADE in simulated, real and spatial-cognition scenarios by 42.0%, 19.0% and 27.7%, respectively, and FDE by 46.0%, 26.6% and 43.5%.

From motion prediction to real manipulation:
Opening a new paradigm for embodied intelligence
WorldFlow’s value lies not only in more accurate motion prediction, but also in exploring a new paradigm connecting world models with robotic manipulation: using objects’ 3D motion trajectories as an interface to turn task intent into a basis for robot-manipulation planning. For example, given “Place the green biscuit box on the translucent box,” WorldFlow can predict the biscuit box’s 3D path and final position. The robot then combines this with its arm reach, joint constraints and grasping method to plan executable manipulation actions.


In physical-robot experiments, the team compared WorldFlow with the motion-prediction model μ₀ chosen for this study. Twelve tasks tested whether the models could give robots capabilities in Pick and Place, Stacking and spatial understanding (Relative Placement). Each method was tested three times per task, for 36 trials in total, with matched initial layouts and shared grasping, planning and control settings. Relative to μ₀, WorldFlow raised trajectory-prediction success from 25.0% to 97.2% and physical-robot manipulation success from 11.1% to 66.7%.
The team proposes exploring object-level 3D trajectories as a planning interface across robot embodiments: different embodiments would share object-motion targets, then use their own grasping, planning and control systems to produce executable actions. Reliable transfer across embodiments still requires separate validation; this is not a control policy proven to work universally on arbitrary robots. From “seeing future frames” to “foreseeing 3D motion,” WorldFlow creates a more direct connection between world models and robot execution, exploring a new technical path toward general embodied intelligence in the real physical world.
About the team
- Yuhao Lu: Co-first author of WorldFlow, with a master’s degree from Tsinghua University’s Department of Electronic Engineering. He currently works on world models and related topics at RoboScience.
- Jiongxu Chen: Co-first author of WorldFlow, with a master’s degree from the School of Intelligent Systems Engineering at Sun Yat-sen University. He currently works on world models and related topics at RoboScience.
- Sheng Wang: Head of RoboScience’s world-model team and lead of the WorldFlow project, with a PhD from the Hong Kong University of Science and Technology. His focus is embodied intelligence and world models, with a research background in vision-language-action (VLA) models and end-to-end autonomous driving.
This article originally appeared on the WeChat account “Synced,” written by Synced.
Related topics
After reading
Join the AINRK community
Read how to join the community and connect with people working in AI.




