Tsinghua survey reviews embodied autonomous systems, data and evaluation through an agent lens

The survey organizes research around perception, planning, execution and feedback, proposes levels of human-machine responsibility, and discusses cloud-edge-device coordination, safety and evidence for cross-task validation.
Multimodal foundation models are expanding the boundaries of robot capabilities. Language understanding, visual perception and action generation are gradually entering a unified learning framework, enabling robots to perform a growing range of tasks.
But as research moves toward open environments and long-term deployment, a deeper question emerges: how can these capabilities be organized into a system that can keep working autonomously?
The real world is always changing. Objects move, contact attempts fail, and task conditions change during execution.
Robots need to make decisions with incomplete information, verify results after acting, adjust their strategies when they drift from the goal, and carry reliable experience into the next interaction. The longer a task continues, the more important this organizational capability spanning perception, decision-making and action becomes.
In response to this shift, a Tsinghua University team revisits embodied-intelligence research from an agent perspective. It places model capabilities, computing infrastructure, safety, data and evaluation within a single system-level picture, discussing technical paths and open questions on the way to greater autonomy.

Project: https://agenticei.github.io/AEI-Survey-page/
From skill learning to a cognitive closed loop
How embodied intelligence reached this point
A continuously evolving theme runs through the development of embodied intelligence: how intelligent behavior emerges through interaction between a body and its environment. From early feedback control to skill acquisition driven by imitation and reinforcement learning, and then deep visuomotor policies, research has continually expanded the behaviors robots can learn and execute. Multimodal foundation models further bring semantic understanding and knowledge transfer across tasks, advancing the development of general-purpose robot policies.

Opening figure of the paper | Traces development from feedback control, skill learning and deep visuomotor policies to general-purpose robot policies and Agentic Embodiment.
Building on this, Agentic Embodied Intelligence (AEI) advances the research question to autonomous organization at the task level. A system needs to understand goals, proactively obtain additional information, arrange subtasks, invoke skills and use physical feedback to decide whether to continue, retry, replan or seek help.
This perspective also provides common coordinates for discussing different technical approaches. VLA, world-action models and programmatic skill composition can perform different functions within the closed loop, and their value must be examined in terms of how they support complete tasks. General-purpose policies provide an increasingly rich repertoire of behaviors; AEI goes further to ask how these capabilities can be organized effectively throughout sustained interaction.
Putting the model back into the system
Rethinking where autonomous capabilities come from
Figure 1 of the survey presents a system-level framework. The model handles embodied understanding, planning, action and reflection. Infrastructure connects skill invocation with robot control. The robot body executes actions in a physical or simulated environment, and new observations and states return to the model. This online interaction forms the operational core of an autonomous system.

Paper Figure 1 | System relationships in Agentic Embodied Intelligence. Online interaction connects the model, infrastructure, robot body and environment. Data and evaluation support capability improvement, while safety and trustworthiness requirements run throughout the system.
Around this core, the environment continually generates interaction experience, data support model training, and evaluation exposes capability gaps and failure modes that in turn influence the direction of improvements to models and systems. Safety requirements span model checks, runtime protection, control restrictions and human supervision.
This means that capability improvements for deployment require coordination across multiple layers. High-level reasoning and low-level control must share sufficiently accurate state information. Skill interfaces need to return execution results and exceptions. The division of work among on-device, edge and cloud resources must meet latency constraints, while data and evaluation must cover the difficulties that occur in real operation.
Only through these connections can progress in one part become more reliable behavior by the system as a whole.
The measure of autonomy
is how much task responsibility a robot can assume
When “autonomy” describes different systems, a necessary question is what work humans still do. Who breaks down the task? Who monitors the process? Who takes over after a failure? Who decides that the task is complete?
The survey uses a conceptual L0–L5 framework to discuss changes in human-machine responsibility. From continuous teleoperation at L0, assisted skills at L1 and controlled task autonomy at L2 to conditional autonomy at L3, humans gradually shift from directly controlling actions to supervision and intervention as needed. L4 goes further toward sustained task execution, outcome verification and failure recovery in a defined domain. L5 points toward general autonomy in the open world.

Paper Figure 2 | Characterizing autonomy levels through the distribution of responsibility between humans and machines. L4 and L5 are listed as research goals in the survey, requiring evidence of sustained operation, recovery capabilities and recognition of capability boundaries.
This framework examines “what can be completed” together with “how much human support is needed.” Whether a robot can recognize the limits of its own capabilities, stop safely in abnormal situations or request help also becomes part of autonomous capability.
For research, it provides a yardstick for analyzing system progress, extending evaluation from task coverage to the responsibilities involved in completing tasks autonomously.
The online cognitive closed loop
Physical interaction changes the conditions on which the next decision depends. In a partially observable environment, a robot acts on limited information, and its actions in turn change the environment's state and subsequent observations.
As the task horizon lengthens, early errors in state estimation may gradually accumulate, leaving a previously feasible plan without the conditions needed for execution.
Examining embodied intelligence from an agent perspective, the survey summarizes the model core of Agentic Embodied Intelligence (AEI) as four coupled functions. It examines how systems use physical feedback to keep correcting goal-directed behavior.
Perception and embodied understanding map task intent to entities, spatial relationships and physical properties in the environment, providing decision-making with an estimate of the current state and its uncertainty.
Reasoning and planning use this information to break down long-horizon goals, compare candidate plans and represent subgoals as steps that robot policies can execute and the system can verify. When observations are insufficient to judge whether an action is feasible, actively acquiring information can itself become part of planning.
Action execution turns subgoals into movements constrained by the robot's dynamics and safety requirements, and returns the actual changes in state.
Closed-loop evolution uses execution evidence to compare expected and actual consequences, identify where a deviation arose, and then adjust actions, revise plans or update experience.
These four functions may be handled by a unified model or distributed across multiple modules. Different architectures face the same interface questions: what information needed for planning does perception retain, and what evidence does execution return that is sufficient to change the next decision?

Paper Figure 3 | The online cognitive closed loop. Perception provides the state information for planning, execution produces observable physical consequences, and outcome verification drives subsequent decisions.
Recovery within a task and updates across tasks require different standards of verification.
Within a single task, the value of reflection can be tested through subsequent intervention: under the same retry budget, can the revised action plan effectively restore execution?
To go further and claim that the system has gained reusable experience, the temporary context of the current task must be removed and the benefits tested on new tasks. When policies or models are updated, the retention of existing capabilities should also be evaluated.
The cost of resetting physical interactions, ambiguity in sensor feedback, and differences between simulation and real deployment all make it harder to attribute failures to their causes.
The survey therefore links update targets to evidence requirements: control adjustments can be verified within the current episode; persistent-memory and skill updates require evidence across tasks; and broader system modifications require independent version comparisons and deployment monitoring.

Paper Figure 7 | Closed-loop evolution in digital and embodied agents. Both kinds of system use execution feedback to locate deviations, but differ substantially in their verification environments, reset costs and deployment conditions.
System support and safeguards
The online cognitive closed loop describes how an agent makes decisions. The system layer determines whether those decisions can be executed promptly and corrected safely in a physical environment, while leaving evidence that can be independently examined.
The survey consequently brings computing infrastructure, safety mechanisms, interaction data and evaluation protocols into the same analytical framework.
Cloud-edge-device coordination and real-time execution.
Embodied systems run closed loops at several timescales simultaneously. Low-level control and action policies face strict response deadlines, whereas long-horizon planning can use larger compute budgets. How inference is distributed among the robot body, on-site edge servers and the cloud depends on each part's requirements for compute, energy consumption and latency.
For fast control, average latency is especially liable to conceal problems. An occasional long delay or disconnection may cause returned instructions to be executed on the basis of outdated state information. Hybrid deployment must therefore also coordinate state synchronization, instruction validity periods and local takeover mechanisms.

Paper Figure 9 | On-robot, cloud and hybrid deployment. On-robot resources face power and heat-dissipation constraints, cloud deployment is constrained by communications, and hybrid deployment must also divide computation while keeping state consistent.
Safety and trustworthiness.
Safety constraints need to span perception, planning and control. Risks identified during perception, preconditions set during planning and restrictions imposed during execution must refer to the same object and rely on state information and authorization that remain valid.
After the task ends, the system must also check for residual risks. If experience will be written to memory or used to update a policy, safety judgments should extend to subsequent tasks, testing whether the original constraints continue to hold in the new version.
Embodied experience for agents.
The survey proposes Agentic Embodied Experience, organizing interaction records through goals, context, decision rationales, subgoals, affordances, motion, feedback and reflection.
This representation links the information available at decision time to the movements that actually occurred and their physical consequences. Researchers can thereby trace where a deviation first appeared and whether subsequent intervention changed the result.

Paper Figure 13 | From sources of embodied data to representations of interaction experience. The data pyramid describes the scale and specificity of different sources, while the experience representation describes how a decision is linked to action, feedback and revision.
Data distribution also depends on the agent's own behavior. Exploration strategies, the timing of requests for help and methods of recovering from failure all affect which states are recorded. Feeding capability gaps exposed by evaluation back into task selection and data collection, then testing the benefits of updates on independent tasks, can form a data cycle driven by capability diagnosis.
Evaluation for autonomy.
Starting from the cognitive capabilities tasks require, the survey identifies five kinds of evaluation scenario: goal disambiguation and active understanding; tool discovery and affordance reasoning; handling online disturbances and recovery; long-horizon tasks in open environments; and social interaction and multi-agent collaboration.
These describe the sources of task difficulty. The earlier autonomy levels address a different dimension: within given operating boundaries, how much responsibility humans and systems each assume for interpreting goals, making decisions, monitoring execution and handling exceptions.
To infer autonomous capabilities from experimental results, evaluation protocols need to specify which physical and semantic states the simulation environment preserves, what skills the tool interfaces provide in advance, and what kinds of human intervention are permitted during operation.
Only by jointly reporting task outcomes, decision quality, execution quality, the recovery process and safety performance under these conditions can one determine which agent capability produced the observed gains and the environments to which the conclusions apply.

Paper Figure 15 | Five kinds of cognitive requirement in embodied benchmarks. Different task categories make different demands on active information acquisition, management of intermediate states and adaptation to environmental change, and correspond to different measurement blind spots.
Closing thoughts
General-purpose policies expand the range of tasks robots can execute. To move toward greater autonomy, research must also explain how these capabilities keep working together in uncertain environments and when systems should recover, seek help or stop.
The paper treats sustained task autonomy within a defined operating domain and autonomy in the open world as research goals at different levels.
Progress should be judged by whether, under explicit operating scopes, task durations and conditions of human support, a system can continue to assume the corresponding decision-making responsibilities and provide evidence of its behavior and subsequent updates that can be checked.
References:
https://agenticei.github.io/AEI-Survey-page/
This article comes from the Xinzhiyuan WeChat account. Author: Xinzhiyuan.
Related topics
After reading
Join the AINRK community
Read how to join the community and connect with people working in AI.




