Ropedia releases Xperience 2 to align real scenes with first-person behavior

Scene geometry, video and body trajectories share one coordinate system. Demos cover simulation conversion, retargeting, virtual teleoperation and trajectory generation; benefits for real-robot training need further tests.
On November 9, Anthropic released Claude Fable 5.2. It reportedly scored 64.9% on the RoboDojo robotics-control benchmark, more than twice the previous Fable 5.1's 27.6%. Just two months earlier, GPT-6 Astra had topped that leaderboard with a 22.48% average success rate, surpassing all 40 public policies.
General-purpose large models' embodied capabilities are improving at a remarkable pace, but leaderboard scores are not the same as real-world capabilities. In a joint study [1], GPT-6 Astra was deployed on eight real-robot configurations: all 11 attempts to insert a plug failed, and only nine of 23 attempts at long-horizon navigation and object search succeeded. The researchers judged that it had preliminary embodied capabilities, but remained some distance from practical use.
How can general-purpose large models control robots better and continuously complete longer, more complex tasks in real environments? Models need both behavioral demonstrations to learn from and environments in which they can perform tasks and accumulate experience. Large models have scaling laws; where are the scaling laws for physical intelligence? Answering that first requires clarifying what kinds of environment and behavior data should be scaled, and how they can be produced and used continuously.
Recently, Physical AI startup Ropedia officially released Xperience 2, offering its own answer: Ropedia Xperience 2 is a real-world environment and behavior dataset for Physical AI. It combines real-scene scans with recordings of human first-person activities, aligning scene geometry, video, trajectories and body poses in the same metric coordinate system.
Using these data, Ropedia converts human movements into robot trajectories and scanned scenes into interactive simulation environments. With this integrated environment-and-behavior data—captured in the real world, precisely aligned, diverse in coverage and reusable—Ropedia hopes to establish a “gold standard” for the next generation of embodied data, providing the data foundation for scaling laws in Physical AI environments and behaviors in the post-GPT-6 era.
- Technical report: Ropedia Xperience 2: A Gold Standard for Scaling Physical AI Environments and Behaviors
- Report: https://arxiv.org/abs/2611.17060
- Project homepage: https://ropedia.com/xperience/2-research
- Dataset: https://huggingface.co/datasets/ropedia-ai/Ropedia-Xperience-2

1,812 scenes form the X in Xperience.
“Gold-standard” data for physical intelligence: environments and behaviors are both essential
Scan, record and align: giving every action a real-environment reference
Xperience 2 retains a corresponding scene reconstruction for each task video. The camera trajectory, hand movements and full-body motion in the video are placed in the same 3D coordinate system and aligned with the room's geometry and layout. Users can review how a person acts in that room and continue using these records to construct robot tasks.
Consider putting a cup in a cupboard. Video can record picking up the cup, opening the door, placing it inside and closing the door, allowing a model to learn the actions and their order. When a robot actually performs the task, it also needs to know where the cup is, which way the cupboard door opens, whether its arm can reach, and how to judge completion. These details must be determined using the scene's geometry and object states. Video alone does not give the robot an environment in which to try the task.
A simulation scene alone is not enough either. Simulation gives the robot somewhere to practice, while human demonstrations can provide a way to get started. Without demonstrations, where to grasp the cup, which way to open the cupboard door and how to place the cup may all require repeated exploration, along with corresponding reward design. The more steps a task has, the more this training often costs.
Existing datasets usually emphasize one side or the other. ScanNet++, Realsee3D and DL3DV-10K provide 3D scene reconstructions and camera poses without corresponding human first-person task videos. EPIC Fields, Nymeria and HD-EPIC record human activities and camera poses, but mostly do not provide scene meshes.
Xperience 2 brings scenes and human demonstrations together. Once aligned, human movements have an explicit spatial reference. They can be converted into motion trajectories according to the robot's body structure and replayed in the same scene for inspection. Original videos can therefore continue to support research on robot movements.
Once a scene has been configured for interaction, the cup position, cupboard-door state and robot starting point can be set, allowing the robot to perform tasks inside it. Missing the cup, placing it on the cupboard's edge or forgetting to close the door can all be judged and recorded from object states. After failure, reloading the scene allows another attempt; changing the initial conditions enables further collection of new trajectories.

Data layers after video-to-scene alignment: meshes, 3DGS, first-person video, head trajectories, poses of both hands and the full body, and activity annotations at three timescales, all in the same metric coordinate system.
This collection builds on Ropedia's earlier first-person data work. Xperience-10M contained around ten million real-world interaction clips and 10,000 hours of first-person video and audio. Xperience 2 adds scene scanning and spatial alignment to these behavior records, giving every activity a corresponding real environment. A single scan can serve multiple task videos recorded in the same scene.
The two collection steps use different equipment. A handheld lidar scanner scans rooms from multiple angles, filling in areas that a person's viewpoint does not easily cover. During task recording, operators wear Ropedia's HOMIE Headset 2, leaving both hands free to manipulate objects. Its four global-shutter fisheye cameras, inertial sensors and four-channel audio jointly record activities; two forward-facing cameras capture the hands and objects being manipulated. The entire collection process needs no fixed motion-capture infrastructure and can take place in homes, restaurants, shops, offices and outdoors.

First scan the scene, then record tasks one by one. The left shows the two steps. On the right, the scanner's path and camera poses, scene reconstruction and head trajectories from the HOMIE Headset 2 recordings appear in sequence.
Operators first scan a room from multiple angles, producing a scene mesh at real scale, a 3D Gaussian reconstruction (3DGS), and corresponding images, camera poses and a sparse point cloud. These data share a gravity-aligned coordinate system. Physical properties and interactive parts can then be configured for the objects, making the scene usable for simulation.
After scanning, the operator puts on HOMIE Headset 2 and completes each task assigned to the scene. Each video begins before the task starts and continues until after completion, retaining a full activity sequence.
The recorded videos are then processed automatically to obtain camera-motion trajectories, depth and estimated poses of both hands and the full body. Video, inertial measurements and audio are aligned in time. Actions receive three layers of language annotations: what the entire activity is, when each subtask occurs, and the states of objects and hands in individual frames.

Estimated full-body poses and trajectories at different times in the same scene, alongside the corresponding first-person views. The data layers provided for each video are listed below.
Next comes alignment between video and scan. The system first locates video frames within the scanned scene, then transforms camera trajectories, hand movements and full-body motion into the scene coordinate system. Registration is followed by checks on held-out validation frames, gravity direction, and agreement between depth and the mesh. Each recording is labeled PASS, WEAK or FAIL so users can filter it.

Video registration: a recording first appears in its own coordinate system, then is registered to the scene coordinate system. The cards at upper right list three checks; the recording is marked PASS only after passing all three.
The processed trajectories and poses are retained alongside the original footage, allowing users to review the video and check whether a particular action is suitable for their training task.
From bedrooms to streets: bringing real-world diversity into the data
From bedrooms and kitchens to offices, shops and streets, the current release contains 1,589 scenes spanning 12 kinds of real spaces. Residential interiors account for 1,389 scenes, or 87.4%; bedrooms and living rooms together account for 976, or 61.4%. The overall collection covers 596 locations, including 443 complete homes. Location counts describe the overall collection and are calculated separately from the number of scenes in this release.
Across the overall task collection, 74 tasks and 4,230 videos were recorded, covering five activity categories: picking and placing, cleaning, tidying, folding and inspection. Grouping by the main verb gives seven kinds of action, including 25 tasks involving putting objects in or taking them out, 23 involving wiping and cleaning, and 18 involving placement and organization. Of the 74 tasks, 31 come from work and retail settings such as restaurants, shops and offices.

High-precision data: making models' spatial capabilities testable
A robot entering an unfamiliar room must reconstruct its surrounding 3D structure, locate objects and understand distances and directions. Xperience 2 builds three benchmarks around these capabilities using the same set of 20 real scanned scenes. Scan geometry, camera poses and spatial annotations provide references that allow specific checks of model capabilities in real environments.

Representative model comparisons on the three benchmarks, with four models shown for each. Lower camera-trajectory error is better; higher object-localization precision and spatial-question-answering accuracy are better. The geometry evaluation uses medium input, 11–100 frames.
Reconstructing space: leading models show reconstruction potential
Recovering a room from images requires reconstructing both scene surfaces and camera motion. In a comparison of 11 models, π³ has the lowest trajectory error and highest surface-reconstruction F-score across three input regimes. DA3-Giant leads in pose recovery with medium and dense inputs. With scan geometry and camera poses as references, errors in depth, trajectories and surface reconstruction can be measured separately. A richer range of real scenes can also help subsequent training and evaluation cover more layouts and observation conditions.
Locating objects accurately: real environments remain challenging
After recovering the room's structure, the model still needs to locate objects. This evaluation requires identifying a target and predicting its 3D bounding box. The best performer, Boxer, achieves 24.69% AP@0.25 and 6.09% on the stricter AP@0.5. Precise localization in real environments remains difficult, highlighting the need for more open real-scene data: broader coverage of layouts, occlusions and object differences in training and evaluation, with reliable 3D annotations for models.
Understanding space: strong performance on spatial questions
The spatial-reasoning benchmark contains 500 questions covering translation, rotation, distance, depth, height and relative position. Gemini 3.8 Flash achieves 74.2% overall accuracy, and GPT-6 Astra (High) 71.8%. Both exceed 90% accuracy on height and relative-position judgments and can already answer several kinds of spatial question well.
The next step is to make this spatial understanding support actual action. Models need to use judgments about position, direction and object relationships for navigation, manipulation and continuous completion of long-horizon tasks. By connecting real environments with human behavior, Xperience 2 provides a data foundation for learning “how to complete a task in a space like this.”
From real scenes to virtual training grounds: expanding ways to generate robot experience
The same real scenes and human activities can support different kinds of robot-data production. The technical report describes four uses: Real-to-Sim turns scanned scenes into interactive simulation environments; human action retargeting converts demonstrations into robot-motion references; virtual teleoperation collects new interactions in simulation; and long-horizon trajectory generation uses programs to complete multiple steps continuously and produce full trajectories.
Scenes and videos retained from one real-world collection can be reused throughout these processes. New robot movements and interaction records continue to be added beyond the original footage.

Data sources, benefits and cost per collection for the four demonstrations, with real-robot teleoperation as the reference. The squares are qualitative ratings from the technical report, not measured values.
Real-to-Sim: building interactive task environments
Real-to-Sim uses geometry and appearance from scene scans, first reconstructing objects individually and then adding collisions, joints and physical properties so they can be manipulated in simulation. In a demonstration based on a real home, the team converted 107 objects, 73 of them interactive.
The converted scene retains the scan's scale and coordinates, allowing it and the scan reconstruction to be rendered from the same camera viewpoint and compared pixel by pixel. In the illustrated view, the median distance between simulated surfaces and the scanned mesh is around 2.5 centimeters, with about 79% of pixels showing a distance of no more than 5 centimeters.

Real-to-Sim, part one: object reconstruction. Top: scan reconstruction, pixelwise distances between the two, and the simulation scene, viewed with the same camera. Bottom: 3D object boxes labeled by physical type after conversion.
The team also ran a comparison in another home, asking GPT-6 Astra to build a scene solely from the collected images, without providing 3DGS. The resulting objects differed in color, size and position. The 3D boxes of the television, refrigerator, microwave and kettle also deviated from their positions in 3DGS. This comparison uses 3DGS as its reference and measures how consistently the two scenes match the scanned reconstruction.

In another residential scene, 3DGS, the scene built by GPT-6 Astra from images alone, and the Real-to-Sim scene are shown from the same viewpoint. 3D boxes mark four matched objects, measuring consistency with 3DGS rather than independent ground truth.
Once objects are modeled separately, their behavior can be configured for tasks. The 73 interactive objects in the demonstration include nine articulated objects, such as refrigerator doors, cupboard doors and drawers; eight soft bodies simulated with finite-element methods, such as cushions; and four pieces of cloth, such as a television cover and towels. Switches and remote-control buttons can be pressed, and faucets can release water. All seven interactions shown below come from the same physical simulation.
Consider putting cups away. Only after the cup, cupboard body and cupboard door are modeled separately can opening the door, placing the cup inside and closing the door be configured. The configured scene can be loaded repeatedly, used for different tasks or given different starting points. The same real scan can therefore continue to serve multiple rounds of robot experiments.

Real-to-Sim, part two: simulated object interaction. All seven interactions come from the same physical simulation.
Action Retargeting: giving robots a starting point from human demonstrations
Human action retargeting uses body and hand movements already aligned to the scene. When a person turns while reaching, a wheeled humanoid robot may need to move its base, turn its torso and adjust its arms and dexterous hands simultaneously. Using the robot's structure and motion constraints, the system computes trajectories for each part, then replays them in the scanned scene. A human task video can thus continue to support robot-motion research without a robot being present at the collection site.
Trajectories must also pass checks for speed, torque, stability and collisions: joint speeds and torques stay within 90% of their limits, the zero-moment point remains inside the support polygon, and collisions with the scene or the robot itself are avoided. Head and fingertip tracking errors are recorded too.

Human action retargeting: operator movements from HOMIE Headset 2 video, shown as a blue human body, are converted into a wheeled humanoid robot's movements and replayed in the scanned 3DGS.
The demonstration uses four complete videos from the same scene: putting down a parcel and operating an air-conditioner remote; washing clothes; putting dishes away; and folding towels and sweeping. Their original duration totals 10.3 minutes. After the videos are slowed to 0.5× for retargeting, the robot base travels 96 meters in total, with movement occurring in 35%–41% of the frames in each recording.
Across the four trajectories, median fingertip tracking errors range from 0.40 to 0.55 centimeters, and all 95th-percentile errors are below 2 centimeters. Every frame passes the constraint checks. Peak joint speed and torque reach 90% and 89% of their limits, respectively, while the zero-moment point remains at least 0.46 centimeters from the support-polygon boundary.

Retargeting results for four recordings in the same scene, labeled with median fingertip errors for each. Orange marks the operator's hand keypoints; gray marks the robot.
This step produces a motion reference for the robot. Replay takes place in a 3DGS scene, where objects do not move in response to the actions. Whether the robot can actually grasp the cup and place it stably in the cupboard still needs to be tested in an interactive simulation environment.
Virtual teleoperation: collecting new interaction experience in the environment
Virtual teleoperation connects an operator to the converted simulation scene. The person controls the robot in real time, objects change in response to its actions, and the system records robot observations and control commands while checking command tracking, collisions and joint limits. After a missed grasp, the operator must keep adjusting to the current situation, changing the subsequent actions as well.
After each round, reloading the scene resets it for the next demonstration. The same environment can record repeated executions, different starting states and adjustments after failures. This approach still needs an operator, but removes the work of transporting a physical robot to the site and manually resetting the environment.
Code-as-Policy long-horizon trajectory generation: continually expanding robot experience
As tasks get longer, the robot must continue from the results of the preceding step. If the cup has already been placed in the microwave, it needs to be taken out; after taking it out, the robot must close the door and continue with later tasks. In a joint study led by SenseTime, continually tracking progress across multiple subtasks remained difficult for GPT-6 Astra. [3]
The team sets an eight-step task in the Real-to-Sim home used earlier. A wheeled dual-arm robot picks up a cup from the kitchen counter, opens the microwave, puts the cup inside, closes the door and starts heating, takes the cup out, closes the door again, pushes open the bedroom door and finally puts the cup on the bedside table. The robot must travel all the way from the kitchen to the bedroom and complete the operations along the route without resetting the scene.
In this demonstration, a policy program completes the subtasks and executes actions through a whole-body controller. The program observes objects only through perceptual inputs, without reading object states from the simulator. Changing the initial poses of the robot and cup lets the same program generate complete trajectories from different starting points.

One execution in long-horizon trajectory generation, shown at 16× speed: a wheeled dual-arm robot continuously completes eight subtasks in a simulation scene converted from a scan of a real home.
Across 30 held-out initial states with different robot and cup positions, all 30 executions complete all eight steps and pass the task-rule checks. Acceptance is determined from object states in the simulator, including requirements that the cup not tip over or fall and that appliances and doors be in the required states.
Each trajectory contains 3,869–4,255 control steps at 30 Hz, corresponding to an average 2.2 minutes of simulated time. Generating one trajectory takes 10–16 minutes in a single simulation process. Once the scene and program are ready, initial states can keep being varied and more compute added to generate trajectories without collecting human video again.

Frames and timings at the completion of each subtask in the same execution. Runs from all 30 held-out initial states complete all eight subtasks.
These 30 trajectories come from the same scene, task and policy program, adding complete execution experience under different initial conditions. Once the scene, robot and task rules are ready, more trajectories can be generated for subsequent data production and model research. Environment coverage and behavior experience can also be recorded and expanded separately on this basis.
Cost and scaling: letting more environments support more behaviors
Real-robot collection requires continuing investments in equipment, people and time. DROID collected 350 hours of robot demonstrations in 564 scenes, with 50 operators working over 12 months. [4] Robots must be transported to the sites, operators must control them throughout, and each round must be followed by a manual reset.
Ropedia Xperience 2 moves some of this work up front into scene scanning, conversion and task configuration. Once a scene is built, it can be loaded repeatedly. Movements in recordings can continue to be retargeted, while virtual teleoperation removes the need for physical robots and manual resets. Scenes and behavior records from the same collection can keep serving these different uses.
Long-horizon trajectory generation mainly consumes compute. At the demonstrated 10–16 minutes per trajectory, a single simulation process running continuously for a day could theoretically generate 90–144 trajectories. Simulated and real-robot data differ in fidelity. These paths can supplement real-robot collection, but their actual effects still need to be tested through real-robot experiments.
Collection locations are more flexible too. Operators can arrive with a scanner and HOMIE Headset 2 without first transporting a robot. Multiroom scans also retain the layout between rooms, allowing robots to move and manipulate objects in simulation. The earlier eight-step task extends from the kitchen to the bedroom, and generated tasks and trajectories need not be limited to the original footage.
From the first to the second Xperience dataset: turning data scale into physical intelligence
From Xperience-10M, released earlier this year, to Xperience 2 today, Ropedia follows a clear path: first record human behavior in the real world at scale, then place those behaviors back into the actual spaces where they occurred, and finally turn environments and behaviors together into data for robot learning, practice and verification.
Large-model progress rests on internet-scale text. Progress in physical intelligence likewise needs a data foundation large enough, realistic enough and capable of continual expansion. As general models keep setting new records in simulation, the real bottleneck is shifting toward real-world environments and experience. Starting with Xperience 2, Ropedia hopes to establish scaling laws for Physical AI environments and behaviors in the post-GPT-6 era, making the question of “how data scale becomes model capability” measurable, testable and ultimately answerable in the physical world.
References
[1] An Unexpected Robot Policy: Early Evaluations of GPT-6 Astra on RoboDojo and Beyond
[2] Systematically Exploring the Capabilities of GPT-6 Astra as Embodied Policies
[3] Has GPT-6-Astra Achieved Embodied Intelligence?
[4] Khazatsky et al. DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset. RSS 2024.
[5] Synced: “Physical AI Is Entering the Era of Experience Engineering: Ropedia Focuses on End-to-End Data Infrastructure and Raises Tens of Millions of US Dollars,” August 6, 2026.
[6] Technical report: Ropedia Xperience 2: A Gold Standard for Scaling Physical AI Environments and Behaviors
This article comes from the Synced WeChat account. Author: Synced.
Related topics
After reading
Join the AINRK community
Read how to join the community and connect with people working in AI.




