Tsinghua and Berkeley open-source uni-robot-agent to unify robot tool interfaces

The framework connects models, simulation, and real robots through modular tools. The paper reports multi-model tests, differences in input conditions, and operating costs, and explores streamlined tools and asynchronous execution.
Editorial correction (November 13, 2026) The original wording “released yesterday” has been replaced with the date on the official release page. Scores under different frameworks and sensor conditions are not model capability rankings under standardized conditions.
Robotics is approaching its own Computer Use moment.
Anthropic released Fable 5.2 on November 9, 2026, the date stated on its official page, presenting robot manipulation evaluations under specified settings.

In the RoboDojo Sim evaluation, Fable 5.2 scored 64.9%, far above GPT-6 Astra’s 29.8% in Anthropic’s internal test under the same configuration.

Back in September, MIT researcher Phillip Isola proposed in Robot-Use Agents that future robots connected to the internet and exposing sensor and actuator interfaces could become tools callable by general-purpose agents.
This also fits the System 1/System 2 division of labor in embodied AI, with general-purpose models handling high-level reasoning and task planning while low-level policies execute concrete actions. Models including GPT-6 Astra and Fable 5.2 have provided initial evidence that this approach is feasible.

Two crucial issues, however, still need to be resolved to make it work in practice.
The first is the model. Closed frontier models have demonstrated good robot manipulation capabilities, but researchers cannot directly access their weights, making continued training on robot interaction data even more difficult.
The second is the system. Robots differ substantially in hardware structure, sensors, and control interfaces, while simulated and real environments differ too. If the tool system must be redeveloped for every new robot, cross-environment transfer and large-scale experiments will remain expensive.
RUA therefore needs a unified interaction interface that abstracts away underlying differences between hardware and environments, allowing models to obtain perception information, call control tools, and receive execution feedback consistently.
Recently, a Tsinghua University research team including Ning Ding, Zhiyuan Liu, Mengdi Xu, and Bowen Zhou, working with the team of Pieter Abbeel, a professor at the University of California, Berkeley and co-director of Berkeley Artificial Intelligence Research (BAIR), jointly introduced and open-sourced uni-robot-agent. It aims to establish a unified, modular harness for Robot-Use Agents (RUAs), allowing language models to control different robots and simulation environments through standardized tool interfaces.

- Paper: https://arxiv.org/pdf/2611.01560
- Blog: https://openrua.ai/uni-robot-agent
- Code: https://github.com/Open-RUA/uni-robot-agent
The framework performed well in evaluations. On RoboDojo, the open-weight model Qwen3.8-Flash-Next scored 31.39 with the framework, exceeding the 28.97 previously published by RoboDojo for GPT-6 Astra with RoboProbe. Under the same new framework, GPT-6 Astra scored 51.90.
Across 19 real-robot tasks, Qwen scored 53.50 without task-specific fine-tuning, leading fine-tuned π₀.₅ by 5.75 points and zero-shot Dream-1 by ten points. GPT-6 Astra with the framework scored 80.50.

Beyond scores, the work tested how far models could go in controlling robots, examining failure causes, capability gaps, and operating costs.
The researchers further aim to build a general foundation through which open-source models can connect to different robots, also paving the way for subsequent training and local deployment.
Why are unified interfaces the foundation for cross-environment generalization?
In coding agents, the same toolset can serve different programming tasks. Models use similar interfaces to read files, execute code, and inspect results, reusing previous tool-use experience across environments.
Robot manipulation involves more hardware differences. Even when grasping objects, single-arm and dual-arm robots may have different action interfaces. Models must understand new parameters, coordinate systems, and control rules, making existing tool-calling experience difficult to transfer directly.
uni-robot-agent seeks to bridge these differences with unified tool interfaces.
For example, a single-arm Franka must locate, grasp, and stack blocks; a dual-arm ALOHA must coordinate both arms to hand over objects; and a real dual-arm YuMi must align and insert a peg into a hole. Despite differences in hardware structure, action spaces, and task goals, models can call unified perception, motion, skill, and task-control interfaces, while low-level adapters handle control differences between robots.
In other words, models can follow consistent tool-calling rules. When connecting a new environment, developers mainly need to implement its adapter, reducing changes to higher-level agent logic.

Tool reuse across different environments.
Unified interfaces have a deeper significance too.
If different environments share tool semantics, parameter units, and feedback formats, interaction trajectories generated by robot tasks become easier to combine. Experience acquired in single-arm stacking, dual-arm handovers, and real-robot insertion can feed into the same data system.
This design creates conditions for cross-environment agent training, giving researchers the opportunity to train a single model on interaction data from different robots and learn transferable tool-use and manipulation capabilities.
How is uni-robot-agent designed?
The design has two goals: to provide a unified foundation for cross-environment agent training, and to evaluate general-purpose models’ existing robot manipulation capabilities as objectively as possible.
To do so, the research team divides the system into five layers: Agent, Orchestration, Tools, Services, and Backends.
The Agent layer interacts with the model and manages context and task decisions; Orchestration schedules tool requests; Tools exposes unified operating interfaces to the model; Services manages runtime resources; and Backends connects specific robots, simulators, and model implementations.

This modular structure supports independently replacing models, tools, and robot backends. Researchers can compare models and accumulate structurally consistent interaction data without rewriting the whole system.
Objective evaluation requires conditions as close as possible to real deployment. Simulators can directly provide exact 3D object positions, CAD models, hidden scene states, and even judgments of task success. Such information is usually unavailable directly on real robots. If a model relies on this privileged information, it becomes difficult to judge its actual visual understanding, spatial reasoning, and manipulation abilities.
In the main simulation experiments, uni-robot-agent therefore provides only RGB images, camera calibration, robot states, and task instructions. It supplies no ground-truth depth, object CAD assets, hidden scene annotations, or task-verifier feedback, and permits no external network access. The researchers also restrict shortcuts tailored to evaluation tasks, preventing models from scoring highly through prewritten task answers.
Without this extra information, how can general-purpose models perform complex robot manipulation? The key is organizing the information they receive sensibly and ensuring reliable tool execution.

Key design choices. Each column shows two complementary designs for physical-state consistency, action-execution accuracy, and context management, respectively.
When a robot grasps an object, for example, the model must calculate grasp coordinates from images and arm positions. If an image comes from an earlier moment but the arm state has already changed, the calculation can be inaccurate.
uni-robot-agent binds camera images, calibration information, and robot states to the same moment, reducing localization errors from unsynchronized states. During action execution, request deduplication prevents repeated operations, while arm-motion monitoring detects stalled actions or execution anomalies.
After an action finishes, the system feeds execution results and new environmental observations back to the model, helping it judge whether the task succeeded and whether to adjust the next operation.
To address the time taken by robot actions, the framework also introduces asynchronous execution. Models can continue other work while some tools run, while motion requests for the same arm queue sequentially to prevent conflicting instructions.
These designs incorporate robotics engineering experience into the harness, reducing interference from misaligned information, repeated actions, and missing feedback.
By organizing information properly and providing reliable control tools, uni-robot-agent still enables competitive manipulation by general-purpose language models without simulation privileges or task-specific shortcuts.
Experimental validation: how far can open models go?
To test the unified harness in practice, the team first ran simulation experiments on RoboDojo.
The benchmark contains 42 robot manipulation tasks across five dimensions: generalization, fine manipulation, long-horizon execution, memory, and open-vocabulary instructions. The main experiments include 252 evaluations per group.
The unified harness has a clear impact.

RoboDojo benchmark results. Avg. is the unweighted average of metrics across the five capability dimensions. †GAP (Hybrid) indicates that the approach incorporates π₀.₅.
With the same Qwen3.8-Flash-Next model, RoboProbe scores just 1.90, RPent scores 7.75, and GAP Hybrid, which incorporates π₀.₅, reaches 21.01.
Switching to uni-robot-agent raises Qwen’s score to 31.39, with a 25% success rate. Its overall score improves by 29.49 points compared with RoboProbe.
These experiments show that tool-system design can substantially affect robot manipulation performance even when model weights remain unchanged.
The framework also supports other models. DeepSeek-V4.1-Flash scores 27.11, while GPT-6 Astra reaches 51.90 with a 46.67% success rate.
Different models have different strengths. Qwen and DeepSeek perform relatively better on memory and open-vocabulary tasks than on fine manipulation and long-horizon execution, while some specialized VLA and WAM models have greater advantages on long-horizon tasks.
Real-robot testing further examines capabilities in physical environments.
Using a dual-arm robot, the team evaluates 19 manipulation tasks across five dimensions: spatial understanding, object generalization, instruction following, compositional generalization, and unseen tasks. Tasks include placing test tubes into racks, passing bottles between arms, stacking blocks in a color sequence, wiping a whiteboard, and completing the Tower of Hanoi.

Representative real-robot tasks across the five dimensions.
GPT-6 Astra scores 80.50, including the full 20 points for compositional generalization. Qwen scores 53.50, ahead of task-fine-tuned π₀.₅ and zero-shot Dream-1.

Real-robot evaluation results. Each capability is scored out of 20, for a maximum total of 100.
However, input conditions differ in this comparison. Qwen and Astra use RGB images plus depth information from a RealSense D405 sensor, while π₀.₅ and Dream-1 use RGB alone. Score differences therefore cannot be attributed entirely to the models themselves.
Of greater interest is which external information models actually rely on.
With both depth and camera calibration, Qwen scores 35.89. Removing depth while retaining calibration yields 31.39. Removing calibration as well, leaving only RGB images and the robot’s own state, reduces the score to 9.13.
Astra scores 55.83, 55.04, and 31.29 under the corresponding settings.

Robot manipulation with reduced geometric information. The figure records the task-completion process and final scores with the skill library enabled, across 42 tasks and 771 tool calls.
This shows that models can use camera calibration, multiview observations, and geometric calculations to perform some spatial localization tasks without direct depth measurements. However, performance drops substantially without calibration, and recovering reliable spatial relationships from RGB images alone remains difficult.
Whether a predefined skill library affects evaluation results is another question the researchers examine.
With both the skill library and skill-authoring functionality disabled, Qwen’s overall score falls only from 31.39 to 30.31, retaining 96.6% of its original score. Success declines only from 25% to 24.58%. This indicates that its performance does not depend primarily on prepackaged task skills; standardized perception and motion tools, together with the model’s own reasoning, can already support a substantial portion of operations.

The effect of skill guidance on task performance.
For costs, Qwen’s average API charge per task is approximately US$1.09, DeepSeek’s is US$0.94, and Astra’s reaches US$21.29.

Execution speed also affects robot-agent efficiency. When the overall score first reaches 25, Astra has cumulatively consumed approximately 7.4 million tokens and taken 42.6 minutes; Qwen has consumed 23.5 million tokens and taken 95.5 minutes; and DeepSeek has consumed 49.9 million tokens and taken 99.2 minutes.

To reduce waiting, the team further tests an asynchronous pipeline.
Across five selected tasks, pipeline execution takes less time than serial execution in every case, with reductions of 4.8% to 73.7%. For conveyor-belt grasping, time falls from approximately 59.7 minutes to 15.7 minutes.

When do general-purpose models begin to acquire robot manipulation capabilities?
A unified harness allows different models to be tested under similar interaction conditions and lets us ask a more basic question:
What capabilities must general-purpose models have to perform robot tasks?
The team further compares 13 models from different families, generations, and sizes, finding no simple correspondence between robot manipulation ability and model size or release date.

Robot manipulation across model generations, families, and parameter scales.
In the Qwen family, the tested Qwen3.5, 3.6, and 3.7 models record no complete-task successes. By Qwen3.8, the 27B model reaches an 8.06% success rate, while Flash-Next reaches 25%. Although smaller in parameter count, the 27B model can complete some tasks that older, larger models cannot.
The GPT family also shows generational changes. GPT-5.6 Sol can already complete some robot tasks, reaching 19.17% success; GPT-6.1 Sol rises to 37.50%, and GPT-6 Astra further reaches 46.67%.
These results suggest that robot manipulation capabilities may already have existed in general-purpose models for some time, with improvements potentially related to training-data distributions, combinations of capabilities, and model training methods.
Analyzing failure trajectories from different models, the team identifies four key capability categories.

The figure shows correlations between models’ overall RoboDojo scores and error rates in the following four capability categories. Lower error rates in these four dimensions generally correspond to better robot-task performance.
First is visual understanding. Robots must identify objects, judge orientation, and confirm the state after an operation. Some earlier Qwen models mistook the digit 0 for 9 or judged an upside-down cup as upright, basing subsequent actions on incorrect observations.
In the inspected trajectories, some older models made these errors at rates of 12.2% to 23.8%. No such errors were found in the corresponding inspected samples for Qwen3.8-Flash-Next.
Second is spatial localization. Models must determine a target’s 3D position from images. Without depth information, they can triangulate using multiview observations, but matching the same physical position across viewpoints remains difficult.
The paper finds that even GPT-6.1 Sol has approximately 31.5% of relevant triangulation calls fail geometric consistency checks.
The third category is geometric calculation. Robots must transform positions and poses among camera, world, and arm coordinate systems. Some older Qwen models have error rates of 47.3% to 81.8% in these calculations, falling to 8.2% for Qwen3.8-Flash-Next and further to 1.5% for DeepSeek-V4.1-Flash.
Finally comes embodied reasoning. Models must judge whether a grasp pose is sensible, whether the arm can reach the target, and whether an action risks collision. Even after locating a target, some models repeatedly fail to grasp it because of incorrect wrist orientation.
The team also analyzes relationships between robot manipulation scores and general benchmarks including ERQA, MMMU-Pro, ScreenSpot-Pro, and LiveCodeBench v6, finding varying correlations with visual understanding, embodied reasoning, spatial localization, and coding. Thus, robot manipulation may reflect compositional generalization of foundational abilities from different domains.

The relationship between robot manipulation and general-purpose model capabilities.
Robot manipulation requires successive visual judgments, spatial localization, geometric calculations, and physical decisions. An error at any stage can cause task failure. Improving RUA will require attention to these interconnected foundational capabilities together.
A general framework:
what does uni-robot-agent support?
Besides direct tool calls, uni-robot-agent supports Code as Policy, Hybrid mode (collaboration between VLMs and VLAs/WAMs), and Clean Mode, which retains only basic tools. Different approaches can be tested within the same framework.
Code as Policy lets models control robots by writing code. A program can perform calculations, call actions, and use execution feedback to decide whether to continue.
Using Qwen, the team compares code control with direct tool calling across RoboDojo’s five capability dimensions. In 252 RoboDojo evaluations, Qwen scores 25.60. Code control scores higher on fine manipulation than direct tool calling, reaching 17.40, but still trails overall.

Comparison of Qwen’s scores in Code as Policy mode.
The framework also supports Hybrid mode, allowing VLMs and VLAs to collaborate on tasks. The VLM understands instructions, analyzes scenes, and organizes tasks; the VLA generates action segments from observations; execution feedback informs subsequent judgments.
The team demonstrates this collaboration in simulated environments for platforms including a Panda arm with an Allegro dexterous hand, the Unitree Go2 quadruped, the Unitree G1 humanoid, and the R1Pro wheeled robot. It connects action models including π₀.₅, GR00T, and NaVILA, covering dexterous manipulation, humanoid manipulation, and mobile navigation.

VLMs and VLAs collaborate through uni-robot-agent across different environments.
As models become more capable, some calculations and operating logic currently encapsulated in tools may gradually move into the model itself, potentially making the harness thinner. To explore this, uni-robot-agent provides Clean Mode, retaining only four basic tools—move, hold, finish, and capture_scene—so models complete tasks through observation, calculation, and combinations of actions.
With camera calibration, Qwen and Astra in Clean Mode retain 90.4% and 90.5% of the full-tool configuration’s scores, respectively, while reducing average token usage by 35.7% and 78.8%, respectively. Comparison with the historical full configuration shows that models can maintain most task performance after tools and context are streamlined. For making the harness thinner and improving token efficiency, this provides preliminary evidence.

Token consumption and score comparisons for Qwen and Astra in Clean Mode.
Compatibility also extends to more real robots.
The team has connected physical robots including ARX One and AgileX Piper-X. Low-level adapters handle hardware control differences, while models still use unified interfaces to observe environments, execute actions, and adjust subsequent operations from feedback. AgiBot G2 and Unitree G1 are also being integrated.

Real-robot integration for ARX One and AgileX Piper-X
Integration code and deployment instructions are also open, allowing developers to run uni-robot-agent on the corresponding hardware, switch models, and test their own tasks.
Conclusion
Robot intelligence stands at a new crossroads.
Traditional specialized policies are tightly bound to hardware, expensive to scale, and face generalization and long-horizon challenges. uni-robot-agent instead has general-purpose models control robots directly through unified interfaces, following a distinctly different approach to scaling.
uni-robot-agent is the first open-source result from OpenRUA. Initiated by researchers from leading universities and frontier laboratories, with backgrounds spanning embodied AI and language models, OpenRUA aims to build open infrastructure for agents that act in the physical world.

Team website: https://openrua.ai/
Seen this way, uni-robot-agent does more than provide a robot manipulation framework. It also demonstrates that, with well-organized information and reliable tool feedback, open-weight models can achieve competitive performance in both simulation and real robots.
But this is only the first step. The next crucial question is whether interaction data can be used to update models continually. uni-robot-agent has built the infrastructure; subsequent training and evolution are only beginning.
Reference links:
https://www.anthropic.com/research/project-pilot
https://www.anthropic.com/research/claude-plays-robotics
https://x.com/FredaDuan/status/2116727410972590380
This article comes from “Synced,” and was written by “Yang Wen.”
Related topics
After reading
Join the AINRK community
Read how to join the community and connect with people working in AI.




