AI guides

CAS Institute of Automation and partners survey how experience becomes improvement in agent RSI

By 机器之心1 viewsSource: 机器之心
Survey of recursive self-improvement in LLM agents across Prompt, Memory, Skill and Harness

The paper compares Prompt, Memory, Skill and Harness evolution through a seven-step lifecycle, emphasizing independent verification, regression checks and long-term maintenance.

From Prompt optimization to Harness evolution, this survey traces the development of RSI and uses three questions and a seven-step lifecycle to explain how agents turn experience into continuous improvement.

Imagine this scenario: a coding agent encounters an error when calling an API and discovers that the interface parameters have been updated. After several attempts, it finally fixes the problem.

But in the next task, will it make the same mistake again? Can this fix be consolidated into a rule, a memory or a skill? When it encounters a new problem, can past experience help it find the cause faster and propose a more effective modification?

These are precisely the questions studied in agent Recursive Self-Improvement (RSI).

Recently, the Institute of Automation at the Chinese Academy of Sciences and Meituan, together with Shandong University, Renmin University of China, the CAS Institute of Microelectronics, Shanghai Jiao Tong University, Li Auto and other institutions, released a survey of recursive self-improvement in LLM agents.

Article illustration

  • Paper title: When, Where, and How Experience Becomes Structure: Mechanisms for Recursive Self-Improvement in LLM Agents
  • Paper: https://openreview.net/forum?id=X8epsNeCZk

The survey examines Prompt optimization, Memory evolution, Skill evolution and Harness evolution as four main lines of research, tracing how agents use interaction experience to change themselves. The paper selects 40 representative studies to map the research landscape and compares the specific update loops in 20 studies, bringing methods scattered across different directions into a common framework:

When should something change? Where should the change be made? How can experience become an improvement that has been verified and can be used over the long term?

01 Agents can accumulate capabilities even when model weights stay unchanged

As agents complete tasks, they leave behind records of tool calls, execution results, user corrections and the process of repairing failures. This information can inform their next action, or be used to modify components of the agents themselves.

For example, a constraint repeatedly verified to be effective can be written into a Prompt; information updated by the user can be stored in Memory; a successful operating procedure can be organized into a Skill; or recurring failures can lead to adjustments to tool interfaces and checking logic within the Harness.

Even if the weights of the large model executing the task remain unchanged, the agent's behavior may improve as these external components are updated. The survey focuses on mechanisms of this kind, while also analyzing some methods that incorporate model training.

“Recursive” introduces an additional requirement: retained experience and components can participate in diagnosing, generating or selecting subsequent modifications. For example, the record of a previous repair can help the system locate the cause of the next failure, while existing update experience can help it choose more effective modification strategies.

This takes the research question a step further: beyond making agents better at completing tasks, we also need to study whether they can improve themselves more effectively.

02 Three questions for understanding the keys to agent self-evolution

The paper poses three interrelated questions about how experience becomes instructions, memories, procedures and operating rules that remain effective over the long term.

When: when should something be updated, and when should it continue to be retained?

Task failure, the arrival of new information or changes to a tool interface may all trigger an update. But whether a modification is worth retaining requires further checks. Its performance on subsequent tasks determines whether it should continue to be used, be revised again or be discontinued.

Where: where does the problem lie, and which component should be modified?

The same failure may have different causes: unclear instructions, outdated information, missing operating steps, or a runtime mechanism that lets the agent skip necessary checks. Evidence from execution needs to support the choice of where to make a modification.

Article illustration

These four kinds of components are classified by the role their content plays during execution, and they can work together. For example, the same API rule can be written into instructions, retrieved as a memory, encapsulated as a calling procedure, or turned into parameter validation before the call.

How: how can modifications be distilled from experience, then verified and retained?

Execution logs need to be analyzed before they can become rules or procedures with clearly defined scopes of applicability. Candidate modifications must also be tested for their effects: whether they fix the target problem and whether they affect existing capabilities. Only then can it be decided how to incorporate them into subsequent operation.

Article illustration

Paper Figure 1 | When, Where and How: how experience becomes an agent's structure.

03 Four lines of evolution: how far has research progressed?

The four kinds of components give agents different places to improve. Following the development of each reveals how the research focus has expanded from generating candidate content to using feedback for revision, selection and long-term maintenance.

Article illustration

Paper Figure 4 | Landscape of representative research on Prompt, Memory, Skill and Harness.

Prompt optimization: from searching for better instructions to improving how instructions are generated

Prompt optimization has evolved through three shifts in emphasis.

The initial question was: can an instruction better than one written by a human be found automatically? Early methods answered by generating many candidate instructions, evaluating them on tasks and keeping the highest-scoring version. Essentially, they were searching the instruction space.

Researchers then found that scoring and ranking alone were not enough. Using specific feedback from failure cases to modify instructions could make improvements more targeted. A class of methods therefore emerged that first generated textual critiques or gradient signals from errors, then revised instructions accordingly. Feedback sources also expanded from simple correct/incorrect judgments to richer structured signals such as compiler errors and execution traces.

The latest developments take the object of optimization one level further. They optimize not only “the instructions given to the task agent,” but the prompt of “the agent responsible for generating instructions” itself. First, the optimizer learns to modify instructions better; the improved optimizer is then used to improve the target agent.

This line of research progressively goes deeper at two levels: how task instructions become more effective, and how the way new instructions are produced can be improved.

Memory evolution: from remembering the past to managing the validity of experience

The evolution of memory mechanisms responds to a practical problem: the more an agent interacts with its environment, the more information it accumulates, but not all information should be treated equally.

Initially, memory addressed a capacity problem: the context window could not hold the entire history, so information needed to be stored externally and retrieved on demand. Systems then began to distill reusable experiential rules from successes and failures, rather than merely storing raw records.

As the number of memory entries grows, new problems emerge: information may conflict, and a fact may have become outdated. For example, after a user moves, delivery tasks should use the new address, while queries about past residence still need the old address. This requires more than simply overwriting old memories; the system must manage whom information applies to and when it is valid. Recent research has begun building relational structures for memories, marking their temporal validity and supporting the addition, deletion, modification and querying of entries.

A further issue is that retrieving a memory does not mean it is genuinely useful. The latest work attempts to adjust memory-utility estimates dynamically based on actual feedback from subsequent tasks, so retrieval choices draw on real usage effects rather than semantic similarity alone.

Memory evolution thus covers content, organization and how experience is selected and used.

Skill evolution: from reusing one success to maintaining a usable set of capabilities

A Skill organizes experience into an operating procedure that can be selected and executed. It usually needs to specify the conditions in which it applies, its inputs, its steps and the requirements for its results, enabling an agent to reuse it in similar tasks.

The earliest approach was intuitive: after an agent completed a task, it saved the successful sequence of operations as an executable program, which could be called directly the next time a similar task appeared. This answered the question of “how to reuse a success.”

But once skill libraries came into use, new challenges followed one after another. On the one hand, an individual skill may be incomplete: it may lack a step, fail to handle an edge case or become invalid because the environment has changed. Recent work has begun distilling repair suggestions from successful and failed execution traces to keep revising existing skills. On the other hand, as more skills accumulate, the system must also handle duplication, conflicts and difficulty choosing among them. A failure may come from choosing the wrong skill, a skill missing a crucial step, or a step being skipped during execution. These three cases need to be diagnosed and handled separately.

Some methods have therefore begun training independent skill managers specifically to learn when to add, revise or retire skills. Other work uses ablation experiments, temporarily removing a skill and observing changes in performance to determine whether it is still serving a purpose.

Skill evolution concerns both the quality of individual skills and whether the entire skill library can remain usable over time.

Harness evolution: letting the agent's runtime framework receive feedback too

A Harness can be understood as the runtime framework that organizes the model, tools, context, execution state and checking mechanisms. It determines how information enters the model, how tools are called, how execution continues after errors, and under what conditions results can be delivered.

Previous research has shown that the design of execution interfaces has an important effect on agent behavior: the same model may perform very differently under different Harnesses. But Harnesses are usually designed and tuned manually by engineers. The central advance in recent work is to make the Harness itself a target for experience-driven modification.

An intuitive example: if an agent frequently skips tests after modifying code, the system can automatically add a completion check to the Harness, requiring all tests corresponding to the current patch to pass before the agent is allowed to report completion. This changes neither the agent's instructions nor its memory; it directly changes its operating rules.

Achieving this requires a closed loop: propose targeted Harness modifications from failure traces, screen candidates through regression tests, then decide whether to retain or roll back a modification based on its actual effects in subsequent rounds. Going further, some methods also retain experience from successful modifications, allowing the improver to consult previously effective strategies when making later changes.

This takes research from “improving the framework for task execution” toward “improving the process that produces the next modification.”

04 A seven-step lifecycle: how does an error message become a reusable improvement?

To compare update mechanisms in different directions, the paper breaks the transformation of experience into seven analytical functions that form a lifecycle framework.

Returning to the example of changed API parameters, what role does each of these seven steps play?

  1. Collect | Collection: save the task, call parameters, interface errors and execution results, preserving evidence for subsequent analysis.
  2. Evaluate | Evaluation: judge whether the call achieved its objective and identify the failed outcome and related feedback.
  3. Diagnose | Diagnosis: locate the cause, such as existing instructions still using old fields, and determine which components should be modified.
  4. Abstract | Abstraction: distill a reusable rule from the specific error, such as “different API versions require different fields.”
  5. Update | Updating: turn the rule into candidate Prompt, Memory, Skill or Harness modifications.
  6. Verify | Verification: test whether the new-version call can succeed while checking that valid old-version calls still work, looking for regression problems.
  7. Consolidate | Consolidation and retention: select modifications supported by evidence and record their applicability conditions and version information for use in subsequent tasks and updates.

The distinction is that Evaluate assesses existing execution, while Verify tests candidate modifications. The former helps the system understand what happened; the latter helps it judge whether a change is effective.

The seven steps provide a common language for analyzing and designing update loops. Individual systems may merge, repeat or omit some stages. The paper's comparison of 20 update loops also shows how methods differ in diagnosis, verification and retention decisions.

Article illustration

Paper Figure 2 | Four kinds of components and the seven-step lifecycle, together with the role of retained experience in subsequent execution and improvement.

05 Continuous evolution: the difficulty lies in every round that follows

As agents run over the long term, experience accumulates, and tasks, tools and models also change. The survey goes on to identify several problems that need to be addressed.

First, attribution across components. A single failed tool call may simultaneously involve outdated memories, missing steps and incomplete parameter checks. The system needs to determine the role of each component and establish whether coordinated updates are needed.

Second, long-term maintenance. A rule that works today may become invalid after an interface upgrade; a previously useful skill may gradually duplicate other skills. Continuous evolution requires handling merging, revision, retirement and rollback, while paying the corresponding maintenance costs.

More crucial is independent verification. An increase in task score can show that an update helps under the current setup. To go further and show that inherited experience makes subsequent improvement more effective, one must compare the later modifications produced with and without that experience under the same starting point, tasks, feedback and compute budget, and test their effects on tasks not used in candidate screening.

From Prompt to Harness, this survey presents multiple paths by which agents change themselves through experience and offers a common perspective for comparing those paths.

The three questions determine when, where and how to change. The seven-step lifecycle helps trace how a piece of experience becomes a system component that can be verified, reused and revised further.

Whether an agent can keep evolving ultimately depends on what each update leaves behind, whether this helps subsequent tasks, and whether it provides a more reliable basis for the next round of improvement.

Author information

First-listed affiliation: Institute of Automation, Chinese Academy of Sciences (Guoliang Fan's research group at the Key Laboratory of Cognition and Decision-Making for Complex Systems).

First author: Siqi Dai, a doctoral student at the Key Laboratory of Cognition and Decision-Making for Complex Systems, Institute of Automation, Chinese Academy of Sciences.

Co-first author and project lead: Jingqing Ruan, senior algorithm engineer at Meituan.

Corresponding author and project lead: Bin Zhang, assistant researcher at the Key Laboratory of Cognition and Decision-Making for Complex Systems, Institute of Automation, Chinese Academy of Sciences.

Research institutions: Institute of Automation, Chinese Academy of Sciences; Meituan; Shandong University; Huazhong University of Science and Technology; Institute of Computing Technology, Chinese Academy of Sciences; Li Auto; Renmin University of China; Nanyang Technological University; Shanghai Jiao Tong University; and Institute of Microelectronics, Chinese Academy of Sciences.

This article comes from the Synced WeChat account. Author: Synced.

Related topics

After reading

Join the AINRK community

Read how to join the community and connect with people working in AI.

Community information (opens in a new tab)