AI news

Agnes open-sources agent harness and outlines TTT adaptation and Playbook iteration

By 新智元0 viewsSource: 新智元
Agnes Harness dual-system architecture and controlled strategy iteration

Agnes AI released Agnes Harness, separating LLM reasoning from a decision model, and outlined TTT and controlled Playbook iteration. Nine team-reported system benchmarks and simulation cases do not isolate the harness's gains or establish general real-world reliability.

Editor's verification note: This article introduces the Agnes team's developer preview and research materials, which have not been independently reproduced. OpenAI's questionnaire was a hypothetical survey of self-reported value among 101 researchers, not actual layoffs or a replacement rate. Sys1's TTT section presents a method and evaluation plan without reporting measured TTT gains; decision-model latency and whole-system scores must not be treated as TTT effects. RSI loop figures for inventory intake, baggage rooms and similar cases come from twin simulations and must not be conflated with separately listed real-robot demos as evidence of improvements at physical sites. The nine benchmarks are an interim self-evaluation of specified model/harness pairings. Six leads and the AppWorld tie apply only to the listed comparisons, without isolating the harness's causal contribution. Controlled strategy iteration must retain isolated evaluation, human sign-off, budgets and safety boundaries. Open source does not establish universal reliability or safety certification. OpenAI research-acceleration report and survey methods (opens in a new tab)Agnes runtime framework and preview boundaries (opens in a new tab)Harness technical report and simulation/system evaluation scope (opens in a new tab)Sys1 technical materials and scope of TTT evidence (opens in a new tab)

On November 18, OpenAI published "Research Acceleration: The Jagged Frontier," presenting internal measurements and survey findings.

For the report, it asked 101 of its researchers a pointed question:

If your team could not use coding agents stronger than GPT-5.2 for the next three months, how many resources would you trade to regain access?

In separate questions within this hypothetical survey, the median proportions of team compute and headcount respondents were willing to trade were 30% and 40%. These were neither unanimous answers nor records of actual cuts.

Article illustration

The answers reflect respondents' self-reported valuation of the tools. They cannot be converted into actual layoff numbers or personnel replacement rates.

Agents have now become a lifeline for leading labs, and even for many individuals.

But putting one into the real world to complete enterprise workflows or direct robotic arms requires far more than an intelligent "brain."

At this moment, Agnes AI has open-sourced Agnes Harness, its framework for running and extending agents.

Its main attraction is not just executing instructions, but accumulating experience through practice and continuing to improve:

At the model level, the report proposes TTT, or test-time training, to give the decision model limited adaptation within a request. This Sys1 report provides no measured TTT gains.

At the system level, RSI, or recursive self-improvement, automatically reviews failures, validates new strategies and captures working workflows as Skills and Playbooks.

One lets an agent learn on the spot; the other gives it long-term memory.

Before discussing how it "evolves," though, let us see how it works.

The brain thinks; the cerebellum acts

Start with a simple task: in a simulated environment, a robot dog must find a door inside a room and pass through it.

Hand this simple job entirely to an LLM and it moves painfully slowly.

Every step requires calling a vision interface, taking a picture, packaging it for the large model and waiting on the spot. The model thinks for ages, then replies: "The door is ahead to the left. Turn left 15 degrees."

Turn, photograph, send and wait again. Most of the time goes into requests and waiting.

When people walk, though, the brain does not concern itself with things like "raise the left foot and shift the center of gravity forward." The cerebellum handles it automatically.

Why make AI's most expensive, slowest brain decide how to take every single step?

With Agnes Harness's decision model, the robot dog can quickly judge whether the door is in front of it and whether it is close enough, reducing step-by-step requests and waits. Decision-step latency must not be equated with completing a whole sequence of mechanical actions in milliseconds.

In the team's illustrated simulation case, time on this type of task fell by 32.9%. That is not a universally measured gain across all robotic tasks.

Article illustration

Behind this is Agnes Harness's first core design: dual-system decision-making.

Many agents put the large model at center stage and ask it about everything. Quality may be assured, but it is like requiring the CEO to sign every decision personally: however capable the CEO, the workflow cannot become fast.

Use a lightweight model to save money? It becomes faster, but falls short as soon as things get complex. Speed and accuracy seem mutually exclusive.

Agnes Harness's design divides the work according to task difficulty.

  • The brain—the large language model—handles deep thinking: understanding user intent, decomposing complex tasks, planning and responding to unexpected exceptions.
  • The cerebellum—the decision model—handles rapid reactions. When task state is already clear, it decides directly which tool to call next, where to route a request and how the workflow should proceed.

This "cerebellum" is Agnes Sys1 1.0, the decision model Agnes connects to the framework as a plugin.

With only 9B parameters, it can be deployed on one machine. It specializes in tool selection, workflow decisions, request routing and ordinal scoring, providing subsecond decisions on routine tasks.

This division clearly resembles psychologist Kahneman's System 1 and System 2: one reacts intuitively and immediately, while the other reasons slowly and rationally.

Giving routine judgments to Sys1 reduces large-model calls, waiting and costs.

Article illustration

More importantly, this cerebellum can be replaced at any time.

Sys1 is the default, but developers can substitute another decision model for their particular setting.

Customer service, warehousing and code review require different kinds of "intuition." The framework leaves room for that customization.

But speed solves only half the problem.

Learn it this time; use it next time

Real intelligence should do more than execute instructions. Executing a wrong instruction faster is worse than not executing it.

It must accumulate experience through the rough and tumble of practice and continue improving.

Agnes Harness therefore designs a two-level path of self-evolution, from "model adaptation" to "system optimization":

The model level explores request-local TTT adaptation; the system level iterates versioned Playbooks through controlled RSI. The evidence and validation stages differ between the two.

Level one, TTT: letting the cerebellum learn on the spot

Training and inference are normally two clearly separated stages.

In real business settings, however, data formats vary enormously and niche contexts keep emerging. Knowledge from pretraining alone often does not transfer comfortably.

To address this, Sys1, the cerebellum responsible for frequent rapid judgments, is described as using a TTT (Test-Time Training) mechanism.

Its proposed operation is particularly ingenious:

When an agent faces the current task material, just before producing a judgment, Sys1 would automatically extract information from the present context and build a lightweight, verifiable "auxiliary task."

Under the report's proposed method, the model would perform a small number of parameter updates within a single request. To balance speed and stability, updates would be strictly confined to the model's final RMSNorm scale parameters.

It is like a sniper about to enter the field. Rather than firing solely on theory learned at training camp, just before pulling the trigger he picks up a handful of soil, senses the current wind and humidity, and quickly adjusts his scope.

The TTT section describes adaptation to the current format and context and an evaluation plan. It does not yet report measured gains attributable to TTT or millisecond-level adaptation effects.

The method would limit adaptation to request-local updates and restore the baseline after the request ends; the restoration design itself does not establish stability across all tasks and must not be described as a guarantee against any degradation.

The model retains a keen sense of the task: that is the first level of evolution.

Level two, RSI: making the system more practiced as it works

For the second level of evolution, Agnes Harness gives the system itself the ability to review its work.

Using the Harness's memory mechanisms—an event ledger and state records—RSI acts in the background as a particularly demanding "review mentor."

RSI monitors the agent's task execution, identifying errors and stages where repeated retries cause inefficiency.

After identifying a problem, the system automatically reviews it, analyzes context and equipment state, and works out a new execution strategy, such as changing the order of API calls or adding validation conditions.

The technical report validates cross-task Playbook candidates in isolated twin simulations and sets requirements for human sign-off, trial runs and rollback. It changes strategy data, not model weights, executable safety code or hardware protections.

The central step is retaining experience. Once a strategy is validated as effective, RSI packages the successful workflow as Skills and versioned Playbooks.

This means the system can store workflows that pass its specified validation as reusable Playbooks. Reliability across settings and gains when transferring from simulation to physical sites still require validation; success is not inevitable.

Article illustration

In practice: how it becomes more proficient

The following are twin-simulation closed-loop cases reported by the team, not independent validation of improvements at actual physical sites.

Receiving equipment into inventory may look simple, but details matter: when the scale reading counts as stable, and in what order the steps should run. An error at any stage can cause failure.

On its first attempt, the agent did not understand how to judge scale stability and used the wrong operating sequence. It ultimately timed out, still unfinished after 30 minutes.

RSI then reviewed the attempt, automatically recorded the correct moment to extract the scale reading and distilled an operating manual.

On the next task, the agent followed the validated workflow and completed it within eight minutes.

Article illustration

The next example was the most complex in the entire workflow.

In a baggage-room scenario, the agent had to scan a code to identify the owner of a fragile bag and sort it, while measuring another bag's actual weight and deciding whether it needed to be held.

On the first attempt, scanning and sorting succeeded, but conflicts between equipment operations and poor weighing timing prevented the agent from obtaining a valid reading.

RSI then performed multiple rounds of analysis and optimization, gradually adjusting the device-call sequence, weighing prerequisites and reading-validation method, and storing the working process as reusable experience.

After optimization, the agent measured the bag at 12.60 kilograms, established that it exceeded the marked weight by 4.60 kilograms and triggered the hold condition.

Ultimately, all nine evaluation checks passed, as did three subsequent independent validation runs.

Article illustration

TTT aims to improve adaptation within a request, while RSI aims to accumulate validated strategies. These materials do not support the conclusion that every judgment or every completed task necessarily makes the system better.

Combining the two levels means the agent need not start from scratch every time.

Putting together AI's "complete form"

Putting these pieces together reveals Agnes Harness's overall architecture, almost as though it were modeled on a person.

The large language model is the brain, responsible for deep thinking. Sys1 is the cerebellum, responsible for rapid reactions.

The Harness runtime itself provides memory: an event ledger remembers how far the task has progressed, while Skills and Playbooks remember how things were accomplished previously, preventing details from being forgotten halfway through a long workflow.

TTT and RSI respectively denote a request-local adaptation proposal and a mechanism for accumulating strategies across tasks. They must not be combined and treated as empirically established, continuous autonomous capability improvement.

To move further into the physical world, the agent also needs a body: MHS, or Model Hardware Standard.

Article illustration

Agents are already capable in the digital world. But ask one to move a robotic arm or read a sensor and problems follow: every device has a different interface that needs individual adaptation, and coordinating several devices is more complicated still.

MHS is Agnes's own device protocol, usable for adapting robots, robotic arms, sensors and other equipment. It is not a certified device driver or an established universal industry standard. Device controllers remain responsible for real-time control and physical safety.

Once connected, the brain understands human intent, the cerebellum rapidly judges each step and MHS translates instructions into operations on specific devices, coordinating several devices as well.

Article illustration

In the demo below, once given a body, the agent can send a robot dog to find "WALL-E" and say hello.

From understanding a vague natural-language instruction to reading device state, calling cameras, planning actions and executing on a real robot, the process requires no step-by-step human intervention.

Article illustration

Overall, Agnes Harness consists of four components plus a self-evolution mechanism running through them:

Brain—the large language model: deep thinking, complex reasoning, task decomposition and planning.

Cerebellum—the decision model: rapid reactions, frequent real-time judgments and execution, with other decision models substitutable as needed.

Body—MHS: connecting physical equipment such as robots, robotic arms and sensors so the agent can act in the real world.

Memory—the Harness: remembering task state and execution experience so complex workflows progress steadily without losing track of details.

Self-evolution mechanism—TTT + RSI: exploring request-local adaptation and controlled strategy iteration, without guaranteeing improvement on every use.

An agent with a human-like division of labor can also perform strongly in practice.

Using end-to-end completion of real tasks as the metric, the Agnes team conducted an interim self-evaluation covering nine benchmarks.

The team publishes system self-evaluations on nine benchmarks

Agnes Harness is not merely a paper design.

Focusing on end-to-end performance on real tasks, the Agnes Harness team conducted a rigorous self-evaluation across nine public benchmarks and compared it with selected baselines.

The results are striking:

On six core evaluations—MHS-Benchmark, WorkBuddy Code, AGH-Benchmark, BigCodeBench-Hard V1, τ-bench text-v1 and DeepPlanning—Agnes Harness achieved the highest results among the listed comparison group!

On AppWorld-dev, which tests comprehensive capabilities, it also matched the top-tier Codex system in the reported comparison.

Article illustration

The comparison group included well-known Harness systems such as DeepSeek Harness, Codex and Claude Code.

Two figures highlighted as reflecting the core architecture's strengths stand out:

On complex-task planning in DeepPlanning, the score reached 87.07.

That score reflects the reported combination of model, tools and Harness in this evaluation. Without isolated ablations, a system's total score alone cannot establish the independent causal benefit of its memory mechanism or dual-system design.

On complex coding tasks in BigCodeBench-Hard V1, the pass rate reached 40.69%.

This is the team's self-reported pass rate for that system pairing. It cannot be attributed to the dual-system design alone and is not evidence of universally validated stability across environments.

Particularly noteworthy is the newly released MHS-BENCHMARK.

This standardized test set was built specifically to examine agents' ability to work across software and hardware.

It selects 100 particularly demanding real tasks:

  • Forty real desktop-software tasks: covering 3D modeling, video production and editing, spreadsheets, document processing, web interaction, image and audio editing, and even gameplay.
  • Sixty physical operations tasks across sites: covering machine-tool control in real factories, precision assembly, on-site logistics scheduling, automated inspection, and complex physical-environment control involving water, electricity and cold chains.

In this test spanning a wide range of software and hardware, Agnes Harness's 41% success rate exceeded the listed DeepSeek Harness pairing's 40% and Codex pairing's 36%. But its desktop GUI subset result was 9/40, or 22.5%, below both comparison pairings. The overall score must not conceal this weakness.

Article illustration

However strong the model and however high the score, someone ultimately has to cover the "last mile."

AI's next step

In enterprise AI transformation, the people covering that last mile are called FDEs—Forward Deployed Engineers.

They work on the front line, deploying AI into the different business workflows of each enterprise.

Agnes Harness's plugin architecture seems tailor-made for FDEs:

Model adapters, tools, roles and even Skills retained through RSI can all be packaged as version-managed plugins.

Through MCP, it can connect seamlessly to an enterprise's existing backend ERP or MES systems.

This means each delivery builds a foundation for the next.

Moreover, Agnes Harness being fully open-source offers a path for enterprise auditing and deployment choices, but does not automatically ensure complete enterprise control over all data and dependencies. Who actually controls the agent's capabilities and data?

Public code can be audited, and pluggable models and Playbooks can support reuse. But data processing, rights to assets and external dependencies in an actual deployment still depend on the models, services and licenses used. MHS is the project's own protocol; open source alone does not establish that all hardware barriers have fallen or that safety risks are absent.

Agnes Harness offers a path toward genuinely independent control and decentralized evolution.

If expensive token bills frustrate you and you are tired of rewriting prompts every day to fix recurring agent errors, you might clone the code from GitHub and explore, try and help build this kind of "self-evolving" agent.

AI's next step is not just understanding the world, but adapting to and changing it.

  • Agnes Harness GitHub repository: https://github.com/AgnesAI-Labs/agnes-harness
  • Agnes Harness RSI technical report: https://github.com/AgnesAI-Labs/Agnes-Harness-Technical-Report-Nov-2026
  • Agnes Sys1 1.0 decision-model technical report: https://github.com/AgnesAI-Labs/Agnes-Sys1-1.0-Technical-Report

This article originally appeared in the Xinzhiyuan WeChat official account, written by Xinzhiyuan.

Related topics

After reading

Join the AINRK community

Read how to join the community and connect with people working in AI.

Community information (opens in a new tab)