Vbot releases Exo2EgoPolicy to explore robot data from third-person video

The team demonstrates third-person-to-first-person generation, hand-joint estimation and simulation retargeting, and opens Echo tools. Generated data still need quality checks and real-robot validation.
Editorial correction (November 16, 2026): Generated viewpoints and skeletons are estimates, and simulated forces are not sensor measurements. The test metrics do not guarantee that arbitrary videos can be turned into reliable robot skills.
Many of us have read plenty of novels in which protagonists “body-swap,” but few have imagined that one day AI would do the same.
Suppose AI traveled back into the 1925 black-and-white silent film The Gold Rush and stepped directly into Chaplin's first-person perspective. What would it see when it opened its eyes?
Look down, and your own hands are holding a knife and fork, cutting into the famous boiled leather shoe on the plate. Look up, and you can even glimpse the starving companion with a large beard across from you.

The next second, switch to the protagonist's perspective in Raiders of the Lost Ark. In the dim ancient tomb, the left hand probes a mechanism while the right holds a sandbag and instantly swaps out the golden idol.

And in the dark, cramped underground electrical distribution room in Jurassic Park, a character urgently works the control panel.
In the generated footage, cyan 3D hand skeletons reproduce even the subtle movements of fingers flicking the illuminated switches one after another.

This sequence of “cyber body-swaps” spanning a century all comes from the brand-new, just-released cross-view generative model Exo2EgoPolicy.
The team demonstrates converting selected third-person videos into generated first-person views and hand-skeleton estimates. Occlusion, camera conditions and data distribution affect the results; accurate conversion of arbitrary videos is not guaranteed.
Body-swapping through a century of iconic movie scenes
Keeping track of people even through rapid camera cuts
Even in complex scenes with several people in frame, frequent cuts and large camera movements, its conversion to first-person views remains fluid.
In the famous tango scene in Scent of a Woman, the ballroom is crowded, and the original footage cuts back and forth between wide shots and close-ups.
From start to finish, the model stays firmly locked onto the blind lieutenant colonel played by Al Pacino.
The demonstration shows generated viewpoints and hand skeletons in the dance scene, but how the footage looks cannot prove that all joint and spatial positions are free of error.

Switch to the rain-soaked street in 1952's Singin' in the Rain, where Gene Kelly spins with his head tilted up and a black umbrella in hand.
The first-person view immediately shows streaks of rain falling all around and the rainy nighttime street spinning rapidly as the character grips the umbrella handle.

Since it can freely “body-swap” into Hollywood blockbusters even with rapid camera cuts, everyday household tasks are all the easier.
At an office desk, someone reaches out to put a water bottle from the desktop into a mesh basket. The first-person view clearly tracks even the 3D skeletal trajectories of the fingers as they grasp it and set it down.

Someone casually folds a red T-shirt at an office desk. Feed in the observer-view footage, and the model immediately outputs a chest-level first-person view with fisheye distortion.
In the footage, the 3D skeletons of both hands pinch the clothing's corners and fold it in a rhythm that matches the original movements precisely.

As the person walks to the pantry, opens the refrigerator, takes a drink and unscrews the bottle cap, the generated footage fully preserves the geometric depth of the refrigerator's multiple shelves.
The fingers' skeletal details are equally clear in the first-person view as the right hand opens the door and the left takes the bottle.

As the person picks up the container of heated food from the microwave and turns toward the long table, the first-person view steadily tracks the skeletal pose of both hands supporting the container's edges.
While the person walks, changes in light and shadow across the tabletop and corridor are also natural and continuous.

Humans have been making films and short videos for more than a century, with the camera almost always occupying an observer's third-person viewpoint.
Now, without depth cameras or calibration parameters, Exo2EgoPolicy has overcome the challenge of reconstructing a century of footage from an “observer view” into a “first-person view.”
At this point, you might think this is another new showcase from some large video-model company.
But unexpectedly, Exo2EgoPolicy was built by a Chinese embodied-intelligence unicorn that makes robots: Vbot!
This time, it is tackling the number-one problem that has troubled embodied intelligence for so long: how to achieve both diversity and massive scale in training data.
At the same time, the team also launched the end-to-end general-purpose data pipeline Vbot Echo and the data-collection application Echo App, and opened them fully to everyone starting today!

Project homepage: https://www.vbot.cn/echo/
Whether it is the vast supply of third-person videos online or footage of clearing dishes and making coffee casually recorded with a phone propped beside a table, it can be fed directly into the system.
The robot can instantly “body-swap” into the camera operator's viewpoint and, following "Exo → Ego → Ego Skeleton", calculate three-dimensional joint trajectories and physical contact forces.
The examples show execution on specific robots after data processing and training. They do not mean that a robot can watch any video and immediately acquire a new skill.

Online videos pile up in mountains
But robots cannot consume them
For a robotics company, putting this much effort into teaching AI to “body-swap” into first-person views is about tackling the data-scarcity problem that has long troubled embodied intelligence.
Just a few days ago, Anthropic released Claude Fable 5.2.
The RoboDojo Sim metric reported by Anthropic is an aggregate evaluation result under its specified internal settings. It cannot be presented without qualification as a uniform success rate across all robotics tasks.

It is already obvious that the brain is getting stronger at a rapid pace.
But however smart the brain, the skills in its hands still need to be built up bit by bit with vast amounts of embodied manipulation data.
Vbot's team understands this firsthand.
Founder Yinan Yu was employee No. 001 at Horizon Robotics and its former president of intelligent driving. Co-founder Wei Song was formerly Horizon Robotics' chief architect for its software platform.
Another co-founder, Hailong Qin, was previously chief scientist at Qianli Zhijia and senior director of autonomous driving at NIO. Zhelun Zhao previously headed marketing for Li Auto's first product line.
After a decade of deep experience in autonomous driving, the team quickly discovered upon entering embodied intelligence that robotics suffers from a far more severe data shortage than autonomous driving.
To collect more data at lower cost, the global embodied-intelligence community has tried every kind of approach—teleoperating real robots, ALOHA, UMI grippers and first-person (egocentric) wearable video.

Silicon Valley unicorn Physical Intelligence and Nvidia's recently released EgoScale are both collecting first-person human video at scale.
But whether data collectors have a GoPro strapped to their heads and a UMI gripper in their hands, or wear a heavy exoskeleton, they face the same bottleneck:
Wearable data-collection devices not only raise hardware costs substantially, but also seriously reduce operators' efficiency in real working environments.
The embodied-intelligence industry as a whole still lacks a data solution that delivers both diversity and massive scale!
To break this impasse, Vbot turned to the largest and most broadly sourced video goldmine in the human world: the internet.
Large language models swept the world by consuming trillions of tokens of text already available on the internet.
And in the video world humans have already built, YouTube gains over 20 million new videos per day on average, while TikTok sees more than 100 million pieces of content uploaded each day!
Internet video is naturally diverse and enormous in scale, containing tens of billions of hours of real human experience in cooking, folding clothes and organizing belongings.
Yet faced with this goldmine, robots previously could not take even a single bite.
The first obstacle is a 90-degree viewpoint mismatch. The vast majority of online video is third-person, rather than the first-person perspective needed for embodied intelligence.

The second obstacle is the absence of 3D hand skeletons (Ego Skeleton) and mechanical signals from the first-person perspective.
Humans and robots differ fundamentally in body structure and joint constraints. Without aligned hand-joint trajectories and contact-force data, robots still cannot move.
Viewpoint conversion and retargeting can supplement training data, but generation errors, physical realism, coverage and downstream training effects still need to be validated individually.
That is the context in which Vbot Exo2EgoPolicy and Vbot Echo were born!

Leave the depth camera behind
Directly output 21 hand joints
For AI to “body-swap” across viewpoints through a screen and extract precise actions, missing hardware data and distorted hand shapes are the two hurdles that must be overcome.
Videos downloadable from the internet come with neither depth-sensor data nor camera-calibration parameters.
Even if ordinary video models can generate images, hands and objects are extremely prone to blurring together when they make contact.

To address this, Vbot incorporated three-dimensional geometric constraints and hand-skeleton supervision directly into Exo2EgoPolicy's video-generation backbone.

The model uses a 14-billion-parameter, 50-layer H3 Ref2VA DiT video-generation network as its backbone, with two parallel encoder branches at the input.
One branch uses a frozen H3 Video VAE encoder to extract appearance-reference features from the original video.
The other uses a Qwen multimodal encoder to parse scene semantics and text instructions. Combined with multi-shot stitching training, this lets it keep a firm lock on the target person even when a film frequently changes camera positions.
To help the model truly understand three-dimensional space, the team injects 6D Plücker camera-ray projections during the first half of training, helping it build “spatial scaffolding.”
In the second half of training, the team switches off the stochastic gate at the end of the camera-ray pathway, completely zeroing out the external geometric input!
The technical report shows that removing the noisy external priors actually reduces flow-matching error by 7.9%–11.2%.
At inference time, the model can therefore dispense entirely with external hardware dependencies and reconstruct three-dimensional space directly from an ordinary video alone.
In evaluations on 546 real-scene clips, Exo2EgoPolicy using zero-prior inference comprehensively outperforms the open-source baseline EgoX, which still requires geometric-prior inputs.
It not only improves long-sequence PSNR by 3.25 dB, but also slashes video-generation FVD error by 39.8%!

More importantly, to complete the "Exo → Ego → Ego Skeleton" pathway, the team attaches two core modules simultaneously at the bottom of the DiT backbone:
One is a three-class hand-object semantic mask head; the other is a two-stage hand-geometry readout head, H3 Hand Readout.
The semantic mask head applies a loss weight of 2.0 to the two hand regions and 1.5 to the manipulated object, guiding the model to focus on hand-object contact regions.
The hand readout head first aligns the ACE-Ego-Hand spatial representation at layer 20 of the model.
The readout head estimates 21 keypoints for each hand, or 42 points in total. The number of keypoints is not the coordinate dimensionality: if each point is recorded with three-dimensional coordinates, that corresponds to 126 scalar values.

The reported PCK@5% rises from 11.52% to 20.02%, an increase of 8.50 percentage points and a relative gain of around 73.8%, rather than a doubling.
In previously unseen scenes, effective hand-joint coverage reaches as high as 98.92%, laying a high-precision foundation for extracting downstream robot manipulation trajectories.

Film 19 seconds on a phone
A real robot learns to clear plates on the spot
With the perceptual representations from Exo to Ego to Ego Skeleton connected, the next step is to turn the “hand skeleton” into “force the robot can actually exert.”
This is where the end-to-end general-purpose data pipeline Vbot Echo and Echo App take center stage.
More importantly, this toolchain is fully open to everyone starting today. Anyone can begin using it directly!

There is no need to buy teleoperation equipment or wear an exoskeleton that reduces efficiency.
Ordinary users can open Echo App on their phones and begin low-barrier data collection using the phone's built-in ToF depth sensor or stereo cameras.
In the filmed demonstration, a phone is propped beside a wooden table to casually record 19 seconds of colored plates being moved from the tabletop into a wooden box. Then the user simply taps upload.
Once the video enters the Vbot Echo cloud pipeline, the intelligent cleaning module first automatically removes severely blurred, black and static frames, and scores the footage by action completeness.
Useful segments of reaching for, picking up and placing plates receive 1.00 and are retained. Unproductive segments of standing beside the table idly observing receive 0.00 and are automatically removed.

The cleaned video then passes through Exo2EgoPolicy, becomes a first-person view and hand skeletons, and enters a unified spatiotemporal human-machine-object interaction (HOI) reconstruction module.
The system aligns the scene's 3D point cloud, task semantics and hand skeletons in the same space and time, then incorporates the target robot's joint constraints to perform cross-embodiment motion retargeting.
Simulation estimates contact forces and joint torques from the modeled geometry, physical parameters and contact process. These results are not direct measurements of the actual forces at the scene of the original video.
In a fine-grained espresso-machine operation test, the operator filmed only a third-person view of both hands receiving grounds, grinding and transferring the portafilter.
Using the extracted Ego Skeleton, the dual-arm robot in the simulation environment can synchronously reproduce the entire sequence of coordinated two-handed operations!

In the real-robot tabletop-tidying experiment, training data at multiple levels, checked for interpenetration and trajectory errors, are fed directly to a physical dual-arm humanoid robot.
Standing in front of the same wooden table, the dual-arm humanoid robot needs no human teleoperation. It reaches out with its mechanical grippers, picks up the blue and green plates on the table, and steadily places them in the center of the wooden box.

That is how a 19-second phone video becomes a real capability for a robot.
The pipeline's capabilities extend far beyond tabletop household tasks: it can also handle full-body interactions between two people outdoors on the street.
Two young men on a roadside bench stand up, walk past one another at close quarters, switch places and sit down. Vbot Echo performs a 4D spatiotemporal reconstruction of the entire video.
The reconstructed footage allows free switching between virtual viewpoints A and B, and a single click can also “body-swap” into the first-person view of the man dressed in black to watch the other approach head-on.

Through unified spatiotemporal HOI alignment, the system reconstructs the two men's full-body movements alongside the 3D street in real time, representing the men as cyan and orange digital-human meshes.
These full-body interaction data are then mapped synchronously onto two humanoid robots, driving the real-robot models through the crossing movements.
The same pipeline also handles, in one go, 12 sets of complex real-world scenes, including tree-lined paths, long staircases, obstacle platforms and outdoor vehicles.

The brain is getting stronger
Data generation offers a new path, but quality and scale still require validation
For embodied intelligence to become general-purpose, it must break through the boundaries of data supply, expanding robots' learning experience from a limited number of collection sites into the broad real world.
The old approach of piling on real-robot teleoperation, ALOHA, UMI and wearable devices not only incurs high hardware costs, but also seriously reduces efficiency in real operations.
The model of scaling dedicated data-collection resources can never deliver both data diversity and massive scale to meet the dual requirements of general-purpose robots.
Vbot has brought together Exo2EgoPolicy, Vbot Echo and Echo App in a three-part collaboration to build an entirely new data infrastructure for the industry:
Exo2EgoPolicy accurately converts third-person internet videos—20 million new videos a day on YouTube and more than 100 million daily uploads on TikTok—into first-person material for robots;
Vbot Echo follows “Exo → Ego → Ego Skeleton” and a closed loop with physical simulation to turn video footage into executable robot-training data;
Echo App puts a low-barrier collection entry point with no wearable burden directly into every ordinary person's phone.
The “Fable 5.2s” are training robot brains to become ever smarter, and the human experience that feeds those brains has finally seen a joint breakthrough in scale and diversity.
The ten billion hours of third-person video lying dormant across the web, together with ordinary people's casually recorded phone clips, have now all become a source of continued robot evolution.
This method offers a path to expanding data sources, but it does not establish that the problems of embodied-data quality, scale and transfer to real robots have been solved.
This article comes from the Xinzhiyuan WeChat account. Author: Xinzhiyuan.
Related topics
After reading
Join the AINRK community
Read how to join the community and connect with people working in AI.




