AI guides

EFE Tests Coding Agents on Simulated Robots: 1,156 Tasks and Up to Three Modes

By 机器之心0 viewsSource: 机器之心
Embodied First Exam evaluates coding agents controlling simulated robots.

EFE brings together 1,156 simulated tasks, 35 task sets and 47 robots, with up to three evaluation modes. Controlled experiments and stage annotations for part of the core set examine failures; results depend on the mode, agent framework and single runs.

Editor's fact-check (November 22, 2026): EFE evaluates simulated robots; it does not guarantee deployment on physical robots across all settings. The results in this article for 10 agents and a 456-task core set correspond to the interim project analysis (opens in a new tab); first version of the preprint (opens in a new tab) and live homepage (opens in a new tab) already differ in their model lineups and core-set definitions; the homepage now lists later material including 481 core tasks. Denominators, modes and rerun results from different versions must not be mixed. The rankings in the original article are not conclusions about the current full leaderboard. Task graphs cover only the specified subset of core tasks, and failure-stage annotations were mostly generated and reviewed by agents. They are not human-confirmed root causes for every step of every task. See the article for the limitations of single runs and annotation.

How should we test coding agents that control robots? EFE's answer is to specify the outcome rather than the method, test the same problem in up to three modes, and go beyond scoring to find the reasons for failure.

Testing coding agents is difficult because they have no fixed way of operating. A trained robot policy does one thing: turn images and sensor readings into actions. A coding agent, however, can control a robot directly, write a control program or train a policy before putting it to work.

To test such agents fairly, the tasks, conditions and scoring all need to be redesigned. To that end, a team from eight universities, including Northwestern University and Carnegie Mellon University, built Embodied First Exam (EFE), using one framework to test different coding agents' ability to control simulated robots. The work makes four contributions: three design principles plus systematic analysis.

  1. Tasks should be diverse, going beyond pick-and-place. There are 1,156 tasks, 35 task sets and 47 robots in total.
  2. Measure how success rates change with physical constraints. Each task has up to three modes, varying what information the agent can observe, whether it can reset and whether the environment waits for it.
  3. Provide unified infrastructure for different models, agent frameworks and control methods. An evaluation can be launched with one command, and switching models or frameworks requires changing just two parameters. Agents can control robots directly, write programs or train policies (Figure 1).
  4. Go beyond scores to identify causes. Controlled experiments isolate the effects of different conditions, while breaking tasks into stages locates the step at which each failure occurs.

Article illustration

Figure 1 | A unified framework supporting different models, agent frameworks, control methods and simulators. Evaluation has three modes: privileged, standard and real-time.

  • Paper title: Embodied First Exam
  • Paper: https://arxiv.org/abs/2611.12177
  • Project homepage and leaderboard (see the website for the latest results): https://www.embodied-first-exam.ai
  • Code: https://github.com/mll-lab-nu/embodied-first-exam

1. What Is Tested:

More Than Pick-and-Place

Of the 745 tasks adapted from existing benchmarks, 618 use single-arm, dual-arm or mobile manipulators. To add other robots and capabilities, the team created nine new task sets in three groups:

  • Fine manipulation requiring sustained contact (RoboARC, DexHandExam and RoboPaint): deformable objects, multi-part assembly and tool use.
  • Moving objects and opponents (RoboDynamicExam and two Embodied Olympics task sets): agents must act at the right moment. Most humanoid robot tasks in the exam fall into this group, such as the spike in the video below.
  • Capabilities beyond executing a given goal (iSpatialGym, Embodied Safety Exam and RoboEngineer): independently interpreting goals, judging whether requests are safe and adapting controllers to unfamiliar dynamics.

Article illustration

A humanoid robot spikes a ball over the net (Embodied Olympics - Fun): GPT-6 Astra, privileged mode.

The team selected 456 difficult tasks from the full set of 1,156 as a core set and evaluated 10 coding agents. Difficulty varies greatly between task sets. The following counts include only task sets with at least five tests. Based on the average success rate of five strong agents, Claude Fable 5.2, GPT-6 Astra, GPT-6.1 Sol, Claude Opus 5.5 and Gemini 4 Argon, 12 task sets reached 70% or higher in privileged mode, compared with only one in standard mode. Next, consider tests that none of the five strong agents solved: In standard mode, these made up at least half the tests in seven of 22 task sets. In privileged mode, that was true of just one of 20 task sets, HumanoidBench (Figure 2).

Article illustration

Figure 2 | Success rates by task set. Dots show the individual success rates of the five strong agents. Bars show the proportion of tests none of them solved; black indicates at least half.

Figure 3 lays out every attempt on the core set in standard and privileged modes. Each column is a test. Tests are first divided into two groups according to whether Claude Fable 5.2 solved them, then ordered within each group by the number of agents that solved them. Gray-to-black cells indicate successful attempts; darker shading means a higher cost for that attempt.

Article illustration

Figure 3 | Every attempt on the core set, excluding real-time mode. Standard mode is above, privileged mode below. Rows are agents and columns are tests.

There are tests at every level, from ones solved by all 10 agents to ones solved by none. Difficulty is continuously distributed, allowing the exam to distinguish agents' abilities. In standard mode, the strongest agent, Claude Fable 5.2, solved half the tests, while several of the weakest solved fewer than 7%. Weaker agents also almost never solved problems that the stronger agents could not: In standard mode, the five weaker agents solved only one additional test beyond those solved by the strong agents; in privileged mode, they solved no additional tests.

The core set excludes tasks the team judged easy and tasks already solved by the screening model, GPT-6 Luna. GPT-6 Luna belongs to the same family as GPT-6 Astra and GPT-6.1 Sol, so this filtering may disadvantage that family. Grouping core-set tasks by the stage categories in their task graphs, described in Section 4, shows that object-relation tasks such as placement and assembly are the most common (Figure 4). In total, 99 tasks contain Dynamic, Body or Competition stages, and 86 of those come from the newly created task sets.

Article illustration

Figure 4 | Core-set tasks grouped by stage categories in their task graphs, with an example image for each group. Each dot is a task within a group; a task occupies one dot for each group to which it belongs. This covers 427 tasks with task graphs. Black dots come from new task sets, gray dots from adapted sets.

With the tasks chosen, under what conditions should they be tested?

2. Under What Conditions:

Progressively Tightening the Conditions for the Same Task

For a given task, the initial scene, success criteria and time limit are the same across modes. The differences are as follows:

Article illustration

Comparing results for the same task across modes reveals the effects of tightening these conditions. Take heading a ball 10 times consecutively (Figure 5): In privileged mode, six agents completed all 10 headers. In standard mode, where they could observe the ball only through camera images, all 10 agents failed.

Article illustration

Figure 5 | Heading a ball 10 times consecutively (Embodied Olympics - Fun). Each agent has two rows: S for standard mode and P for privileged mode. Each column corresponds to one header. Six agents completed the task in privileged mode; all failed in standard mode.

Breaking tasks into stages, using the method in Section 4, and comparing failure rates stage by stage between privileged and standard modes shows that fine stages are affected much more than coarse ones. Here, a stage's failure rate is the proportion of attempts reaching that stage that fail there. Among stage types present in at least two task sets, switching to standard mode increased failure rates for fine stages such as assembly, pose alignment and contacting moving objects by 19 to 51 percentage points. For coarse stages such as placement and grasping, seven of nine types increased by only 3 to 18 percentage points (Figure 6, center). For example, the failure rate for assembly rose from 25% to 76%, while placement rose from 18% to 31%. Testing the modes separately reveals which kinds of action are affected by the tighter conditions.

Article illustration

Figure 6 | Left: GPT-6 Astra's success rate as individual privileged capabilities are restored, as discussed in Section 4. Center: Failure rates for each stage type in privileged and standard modes. Right: Among tests GPT-6 Astra failed in its baseline run, the proportion solved under each condition, grouped by whether failure occurred at a fine stage, in black, or a coarse stage, in gray.

With the conditions set, what platform should be used?

3. What to Test With:

Specify the Outcome, Not the Method

EFE fixes only three things: the starting point, the resources provided and how success is determined.

  • Standardized packaging. Each task is packaged in the format of Harbor, the evaluation framework used by Terminal-Bench. Starting an exam requires just one command; switching the model or agent framework requires changing two parameters. Within their own containers, agents can install packages, write and run programs, and train models.
  • Independent scoring. The scoring program runs outside the agent's container, so the agent cannot alter the evaluation result.
  • Remove known answers. Adapted tasks retain the original benchmark's scenes, robots and success criteria, with stricter success criteria in some task sets, but remove its expert policies and demonstration data.
  • Resource limits. Every task has limits on time, tokens, CPU, memory and disk, as well as GPU memory for tasks requiring a GPU. The median time limit is one hour, ranging from 30 minutes to four hours.

Within this framework, agents choose their own control methods, and those choices differ substantially across modes. Across two random samples totaling 1,760 attempts, excluding Claude Fable 5.2 and Kimi 3.1 Pro, 63% of standard-mode attempts used direct control, viewing an image and sensor readings after each action segment. In privileged mode, 79% wrote programs to compute actions. In real-time mode, 97% wrote programs, most of which read sensors and computed actions themselves to form closed-loop control (Figure 7, left). Policies were trained in only seven attempts. Methods also depended on the resources provided: In the soccer-task version with a GPU and training libraries, policies were trained in 25 of 28 tests (Figure 7, right).

Article illustration

Figure 7 | Left: Shares of control methods across the three modes. Right: Numbers of tests in standard and real-time soccer modes in which agents trained policies, scored goals and won. Black indicates access to a GPU and training libraries; gray indicates CPU only.

Because frameworks can be swapped, their own effects can also be measured. GPT-6 Astra showed the largest difference: It solved 34% of standard-mode core tests in Codex, compared with just 27% in Claude Code. Claude Opus 5.5 performed similarly in the two frameworks. EFE therefore lists leaderboard results by model-and-agent-framework pairing.

Completing the exam tells us only whether agents succeed or fail. Improving them also requires knowing why they fail.

4. Systematic Analysis:

Go Beyond Scores to Identify Causes

EFE uses two methods to investigate causes: controlled experiments to isolate the effects of different conditions, and stage decomposition to locate each failure.

Controlled Experiments: Restoring Capabilities One at a Time

Two capabilities available in privileged but not standard mode are the exact states of all objects and unlimited resets. How much does each contribute? Using the standard-mode interface, the team restored them individually for GPT-6 Astra, with one run per condition (Figure 6, left).

Rerunning after restoring a capability is itself another attempt at the exam, and some failures will succeed simply on a second try. To rule that out, the team added a control condition: Restore no capabilities and simply stop telling the agent whether it has succeeded, roughly amounting to another attempt.

Among the 91 tests that failed at fine stages in its baseline run (Figure 6, right), the control solved 19%. Restoring exact states solved 21%, comparable with the control. Restoring unlimited resets solved 36%, with almost all successes coming after a reset or restoration from a saved state. For fine stages, seeing accurately is not enough; agents need repeated tries.

Locating Failures: Breaking Tasks into Stages

The scoring program judges only whether an attempt succeeds, not where it goes wrong. EFE therefore provides a task graph for each of 427 core tasks. Each node represents an environmental change required by the task, such as 'the refrigerator door is open.' Edges specify the order of completion. Each stage is labeled with its type, such as opening or placement, and the contact it requires (Figure 8, left).

Article illustration

Figure 8 | Left: A task-graph example for moving groceries (BEHAVIOR-1K), including stages, ordering, types and contacts. Right: Stage completion over time in three attempts, with red lines marking the failure stages.

Each failed attempt is assigned at most one failure stage. If an irreversible event such as a fall occurs, the stage containing that event is recorded. Otherwise, the stage containing the agent's earliest uncorrected error is recorded. For example, in the grocery-moving task in Figure 8, Claude Opus 5.5 falls while carrying milk. The failure stage is Stage 4, and the failure manifestation is loss of balance. Putting different agents' failure stages for a task together shows where each stopped. In rope threading, for example (Figure 9), the rope must pass through five locations in order: hole, ring, hole, ring, hole. Only Claude Opus 5.5 completed it in standard mode. Claude Fable 5.2, GPT-6 Astra and Gemini 4 Argon all stopped at the second ring, while four of the five weaker agents failed to pass through the first hole.

Article illustration

Figure 9 | Rope threading (RoboARC, standard mode). The rope must pass through a hole, ring, hole, ring and hole in order, for five stages in total. Shown are the stages each agent reached, its failure stage and images at the time of failure.

Coding agents created both the task graphs and annotations. GPT-6.1 Sol in Codex wrote all task graphs and annotated most attempts, including every attempt by Claude Fable 5.2 and by itself. Claude Sonnet 5.5 in Claude Code annotated the remaining attempts and reviewed all annotations. The annotations cover 8,480 attempts across 427 tasks and 857 tests, including 6,027 failed attempts.

Stages make it possible to see where difficulties concentrate. The team examined 192 standard-mode tests, each with at least three stages and at least four failures that could be assigned to a stage. In 51 of these, roughly a quarter, more than 80% of failures occurred at the same stage. Catching a ball twice in succession is one example (Figure 10).

Article illustration

Figure 10 | Catching a ball twice in succession (RoboDynamicExam). The stages each agent reached and failed at, and replay images from three failed attempts in standard mode.

In this task, the robot must catch a thrown ball in a cup, pour it into a chute, wait for it to roll down a ramp and bounce, catch it again, and finally pour it into a moving cart. All 10 agents failed in standard mode. Claude Fable 5.2 got the furthest: It caught the ball both times, but when finally pouring it toward the moving cart, the ball landed beside the cart. Of the other nine agents, eight caught the first ball, then all eight failed on the second catch for different reasons. Some struck the chute or ramp while pouring the ball or moving the empty cup. Some hit the scene with the robot arm while adjusting the cup to receive the bouncing ball. Some never poured the ball out of the cup. In other cases, the ball arrived before the cup had been lowered. Claude Opus 5.5 estimated in its own report that it had only about 1.3 to 1.6 seconds to catch the ball after it left the cup, whereas straightening the cup after pouring took about 1.4 seconds.

Limitations

This evaluation still has limitations. Each test is run only once. In task sets containing only five or six tests, one test can change the success rate by 17 to 20 percentage points. GPT-6.1 Sol, which annotated most attempts, is itself an evaluated agent and annotated its own attempts; it is also in the same family as GPT-6 Astra. Three runs of GPT-6.1 Sol showed that the tests solved by the same agent can vary substantially on another attempt. The team plans to rerun the core tests and provide confidence intervals.

See the paper for the author list.

This article was originally published on the WeChat public account 'Synced,' by Synced.

Related topics

After reading

Join the AINRK community

Read how to join the community and connect with people working in AI.

Community information (opens in a new tab)