SkillMaster jointly learns task execution and skill editing, evaluating downstream reuse

SkillMaster trains one policy to create, update and retain external skills, rewarding changes on related probe tasks. The method is evaluated in ALFWorld, WebShop and search-based question answering.
Growing interest in Recursive Self-Improvement (RSI) has brought greater attention to agents' long-term capability development. Skills distill interaction experience into reusable procedures and task strategies, carrying gains from one task into later ones. Teaching agents to create and improve skills autonomously offers a concrete route toward continued evolution.

Figure 1: Left, an external module manages skills that the agent retrieves and uses. Right, SkillMaster makes skill management part of the agent's own policy, learned jointly with task execution.
Existing methods, however, usually rely on external teacher models, preset rules or auxiliary modules for creating and updating skills (Figure 1, left). The task agent learns to use skills but is rarely trained directly to improve them. Autonomous improvement requires making skill evolution itself a learnable agent capability (right).
Professor Li Yong's team at Tsinghua University proposes SkillMaster. The paper has been accepted at NeurIPS 2026; first author Yang Min is a doctoral student at Zhongguancun Academy.
The starting insight is intuitive: a valuable improvement should make similar previously failed tasks easier to solve and previously successful tasks take fewer steps. SkillMaster turns both effects into learning signals for skill editing, training the same agent to create, update and retain skills from experience. Skill evolution and task execution thus enter reinforcement learning together.

- Paper title: SKILLMASTER: Toward Autonomous Skill Mastery in LLM Agents
- Author: Min Yang, Jinghua Piao, Xu Xia, Xiaochong Lan, Jiaju Chen, Yongshun Gong, Yong Li
- Affiliation: Shandong University, Zhongguancun Academy, Tsinghua University, Southeast University, University of Science and Technology of China
- arXiv: https://arxiv.org/abs/2605.08693
- GitHub: https://github.com/sduyangmin/Skill-Master
Core insight: make failed tasks easier to solve
And successful tasks more efficient
To train agents to improve their skills, it is first necessary to know what a modification brings to subsequent tasks. Using only whether the task ultimately succeeds as the reward makes it difficult to distinguish execution capability from skill quality, and even harder to provide clear feedback on a particular skill edit.
To this end, SkillMaster focuses on the actual effect of skill modifications on subsequent related tasks. If a skill fills a gap in existing knowledge, it should increase the chances of success on tasks that previously failed. If it improves an existing operating strategy, it should reduce unnecessary steps on tasks that could already be solved. These two observable changes provide a basis for measuring the effects of skill modifications.
Method:
Jointly learning execution and skill evolution
Autonomous skill creation, revision and retention require three answers: what evidence motivates an edit, how its value is measured, and how task execution and skill management are optimized jointly. SkillMaster addresses them through trajectory-based skill reflection, downstream utility rewards and DualAdv-GRPO respectively.

Figure 2: SkillMaster's three components: trajectory-based skill reflection, counterfactual skill-utility rewards and DualAdv-GRPO joint optimization.
Learning skill management from interaction traces
The skill-review stage gives the model task goals, retrieved skills, action/observation traces and final feedback. It creates, updates or retains skills using propose_skill, update_skill and keep_skill. A missing search strategy can prompt a new skill; feedback contradicting an existing operational rule can prompt revision.
Task actions and skill-tool calls are generated by the same policy, so skill management is also incorporated directly into model training. SkillMaster first uses supervised fine-tuning to give the model basic task-execution and tool-calling capabilities. It then uses skill-utility feedback to optimize decisions about creating, updating and retaining skills through reinforcement learning.
Evaluating skill utility through counterfactual comparisons
For a candidate skill edit, the system selects a set of related probe tasks and executes them using the original skill library and a temporarily modified skill library, respectively, directly comparing the downstream changes caused by the modification. Whether previously failed tasks become successful, and whether previously successful tasks require fewer execution steps, are both reflected in the utility score, providing direct training feedback for skill editing.
The probe tasks come from the training-task pool and are selected according to task category, product category or the semantic similarity of questions. The system jointly considers the change in the probes' average scores and the numbers of tasks whose performance improves or deteriorates to calculate a utility reward. It uses that reward to optimize skill-editing decisions. During training, only modifications with a positive utility reward are written to the persistent skill library.
DualAdv-GRPO jointly optimizes two decision types
Task execution mainly receives environment-outcome feedback; skill management uses tool-call validity and downstream utility. Because their rewards differ in scale and meaning, DualAdv-GRPO estimates advantages separately, assigns them to the corresponding generated content and updates one shared policy. The model learns both effective task execution and reusable-skill improvement.
Experiments:
From task performance to skill evolution
Experiments use Qwen2.5-7B-Instruct across ALFWorld, WebShop and SearchQA. ALFWorld contains six simulated household-task categories; WebShop tests multistep product search and purchase; SearchQA includes seven single-hop and multihop question-answering datasets.

Table 1: Main ALFWorld and WebShop results.
SkillMaster achieves the best results among compared methods on both ALFWorld and WebShop: 98.7% and 82.0% success, respectively, improving on SkillRL by 8.8 and 9.3 percentage points. Four ALFWorld task categories reach 100%, and the other two exceed 95%.

Table 2: Results on SearchQA's seven datasets.
SearchQA's overall score is 48.0%, highest among compared methods and 0.9 percentage points above SkillRL. Five of seven datasets outperform SkillRL, showing the framework's effectiveness in search-augmented question answering.

Figure 3: Left, component ablations. Center, changes in skill-management operation proportions. Right, changes in library size.
As shown on the left of Figure 3, removing the skill-utility reward, switching to merged advantage normalization, or selecting probe tasks at random all reduce performance. This supports the roles of utility feedback, separate advantage estimation and related-task selection. The middle and right panels of Figure 3 show changes in skill-management behavior and skill-library size, respectively.
In ALFWorld, creation and update operations are more active early, while skill retention rises from about 81.25% to 97.66% later, marking a shift from supplementing and revising toward preserving accumulated skills. The authors also freeze trained parameters and test skill construction during new interactions. Without counterfactual probes or utility feedback, valid management operations execute directly for reuse in later tasks. Starting from an empty WebShop library, enabling both construction and retrieval raises success from 65.63% to 71.88%, suggesting the trained policy can generate and reuse useful skills from new experience.
Case studies:
Creating and revising skills from execution evidence

Figure 4: The agent identifies gaps, corrects rules and validates edits on related tasks using task traces.
As shown in Figure 4, in an ALFWorld cooling task, the agent spent many steps on cabinets and drawers where food was unlikely to appear, and ultimately failed to complete the task. During its review, it identified that the existing skills lacked search-priority rules. It therefore created a new skill to guide subsequent searches toward high-probability locations associated with the target object.
Another example revises an existing skill. ALFWorld's interface requires the agent to hold an object when heating it, while the old skill instructed placing it inside the microwave first. The model corrects the precondition from feedback. On related probes, heating shrinks from eight steps to two, removing six unnecessary operations and directly demonstrating downstream utility.
Outlook:
Exploring RSI by learning skill evolution
Starting from skills as a reusable carrier of knowledge, SkillMaster explores a path toward agents' recursive self-improvement. Agents discover knowledge gaps or inefficient strategies through interaction, change their subsequent behavior through skill editing, and then learn to make more valuable editing decisions based on the actual effects. Task execution and skill improvement are linked, enabling agents both to accumulate skills and to improve their ability to improve skills through training.
The method has been validated in structured interactive environments and search-augmented QA. Open environments require better probe selection and lower-cost utility evaluation, with extensions to skill deletion, sharing across agents and continual learning under distribution shifts. These directions could support joint long-term evolution of libraries and agent capabilities.
This article comes from Synced. Author: Synced.
After reading
Join the AINRK community
Read how to join the community and connect with people working in AI.




