RSI Arena tests autonomous post-training: preference, capability and safety do not improve together

Ten agents trained from the same base model under equal budget limits, then iterated using blind-test feedback. Rating gains could coincide with longer answers, while capability and safety still required separate tests.
Editorial correction (November 16, 2026): The training agents in the experiment helped update the target models, but the trainers themselves were not recursively updated in this setting. Missing words at the end of the original article have been restored.
Under equal compute budgets, simply piling on computation does not guarantee a model's win rate during self-improvement. When models optimize for “popularity,” they are prone to catering to people, which can in turn lead to deterioration in safety metrics. The study introduces the RSIHF framework for the first time, demonstrating that when AI automatically trains AI, a model's “popularity” cannot be equated with its “safety”: the two must be evaluated separately.
If an AI researcher raises evaluation scores, outside observers cannot easily tell whether the work has improved the model itself or found a shortcut suited to the evaluation rules. In some automated research projects already in operation, agents optimize only for a specified benchmark or a fixed evaluation procedure. The resulting scores are convenient for comparisons, but may also encourage overfitting to the evaluation, or even reward hacking: agents exploit loopholes in the evaluation rules to raise their scores without corresponding gains in actual capabilities.
To make these issues easier to assess, Silicon Valley-based AI company Bake AI, together with Hugging Face, Stanford University, the University of Notre Dame and the University of Washington, launched an experiment called “RSI Arena,” livestreaming the entire process. The researchers did not assign the agents a benchmark. Instead, they set a relatively broad task: within limited resources, train a base model into the most capable model possible. The trained models would undergo independent tests as well as blind-test voting by real users.

Figure 1 | Livestream of RSI Arena: AI agents autonomously conduct model post-training, with the experimental process, training curves and compute consumption visible in real time.

- Paper: RSI Arena: Towards Recursive Self-Improvement with Human Feedback
- Paper URL: https://arxiv.org/pdf/2610.33624
- Project website: https://rsiarena.org/
The questions and requests submitted by participating users cover a broad range, with no fixed set of reference answers, making it difficult for models to cater to them deliberately. The task of “becoming the most capable model possible” is difficult to reduce to a single scoring metric disclosed in advance and available for repeated, targeted optimization. Through broad task objectives and open human evaluation, the experiment was designed to make reward hacking against known scoring rules more difficult.
Under this experimental design, ten frontier models acting as agents autonomously completed the post-training workflow, then continued iterating with user feedback. A central focus of observation was how AI would define “better” under this kind of task and where it would direct model optimization.
Experimental setup: the same starting point, with training in two stages
The ten models serving as training agents included GPT-6 Astra, DeepSeek V4.1 Flash and Kimi K3. Every agent trained the same NVIDIA Nemotron 3.5 Lightning 30B-A3B base model, and all training jobs ran on the same 64-GPU cluster.

Figure 2 | RSI Arena system architecture (Figure 2 in the paper). Each agent works in an isolated sandbox and shares the GPU cluster through Slurm scheduling. A management virtual machine handles agent execution, credential isolation, budget accounting and model-submission validation.
The experiment's training was divided into two stages, with all participants given the same initial conditions. Stage one lasted six days. Each agent had 1,000 GPU-hours of compute and $300 in API credit, and had to autonomously complete the entire workflow of data selection, training implementation, evaluation and model submission. The final stage-one assessment included both 20 benchmarks and human blind tests. Stage two lasted 72 hours. Each agent received records of 213 to 276 battles involving its own model, including the questions, both sides' answers and the corresponding votes. Each agent also received another 500 GPU-hours and $150 in API credit to continue iterating on the model.
NVIDIA's official Instruct version, with reasoning mode disabled, served as the common reference in both rounds of evaluation. Human ratings controlled for style factors such as answer length and the presentation of lists. Overall benchmark progress was measured using 12 capability evaluations and represented the fraction of the capability gap between the base model and the official version that had been closed.
Stage one: everyone submits, but more compute does not necessarily win
After stage-one training, all ten agents had produced usable assistant models. After adjustment for style, the model trained by Astra received human ratings close to NVIDIA's official Instruct version. Across the 12 capability evaluations, it closed a median 76% of the capability gap.

Figure 3 | Stage-one human evaluation results (Figure 6 in the paper). The left panel compares ratings before and after style control, while the right panel shows each model's battle results and the amount of feedback. “Anonymous Model” in the figure corresponds to Step 5 Preview.
The training process also revealed a number of engineering problems. GLM-5.3 encountered insufficient GPU memory during training, spent around seven hours troubleshooting through a total of 39 test jobs, and ultimately autonomously implemented a trainer that ran about seven times faster.
Budget management also differed markedly between agents. Grok 4.7 exhausted its allocated API credit early and left the training process with nearly 70% of its GPU time still unused. Gemini 3.8 Flash voluntarily stopped training after using only around 19 GPU-hours. Yet these two models ultimately received higher human ratings than two entrants that had used almost their entire 1,000 GPU-hour allowances.
These results indicate that the agents can already complete autonomous post-training lasting several days. Under compute constraints, good strategies often play a greater role in determining the quality of the delivered model than resource consumption alone.
Stage two: ratings rise, but what exactly improved?
RSI Arena collected Arena-style human feedback both on-site at COLM 2026 and online, providing it as a data source that agents could use in stage two to keep optimizing their models. With NVIDIA Instruct as the reference and after controlling for style, eight of the ten models improved their relative human ratings, with a median overall gain of 28 points.

Figure 4 | Stage-two human evaluation results (Figure 11 in the paper). The right panel uses the same reference model to compare relative ratings across the two stages, avoiding direct comparisons of absolute scores from two independent arenas.
However, the median improvement in overall benchmark progress during this stage was only two percentage points. The correlation between this metric and changes in relative human ratings was −0.19, and the two did not show a pattern of improving together. The model trained by DeepSeek is a typical example: its relative human rating improved by 172 points, while its benchmark progress rose by only four percentage points. Meanwhile, its median answer length grew from 61 tokens to 508.
The agents' choice to optimize answer length was associated to some extent with the stage-one voting results. Stage one had 999 decisive votes in which the two answers differed in length. The longer answer won 72% of those votes. When one answer was more than 6.7 times as long as the other, the longer answer's win rate reached 81%.

Figure 5 | Answer length and human preference (Figure 8 in the paper). The left panel shows the longer answer's win rate at different length ratios; the right panel shows the effect of style control on ratings. The association between length and win rate does not establish causation.
One agent identified length as the “only lever” affecting votes. By modifying prompts and adjusting how answers were generated during training, it steered the model toward longer outputs. Another agent used a reward model to select the best version, only to find that the reward model predicted votes about as well as a prediction based solely on answer length.
Length was not the only factor affecting ratings. The model trained by Step 5 Preview showed essentially no change in answer length, yet its relative rating increased by 98 points. In stage two, agents also discovered and fixed incomplete answers, mistaken refusals of harmless requests, content loops and output truncation. Across all experimental records, no instances of soliciting votes, impersonation or manipulation of the arena mechanism were found.
These results also reveal the boundaries of the experimental design. Even after narrowing the scope for gaming fixed evaluation metrics, it could not entirely prevent agents from seeking patterns in human preferences. Human feedback can identify genuine problems in a model, but it also contains users' preferences about how content is presented. Stage-two training additionally involved more R&D time and resources, so not all changes in that stage can be attributed to human feedback alone.
Does being more popular mean being safer?
The experiment gave agents no explicit safety requirements, and the models' final safety performance diverged. In stage-one tests, three models were more willing than the base model to comply with harmful requests on StrongREJECT. In stage two, the model trained by MiniMax rose from 2.6 to 24.5 on this metric, indicating increased compliance with harmful requests. Some other models, meanwhile, improved their safety on the same metric.
Across the experiment as a whole, the three kinds of metrics—preference, capability and safety—did not consistently move in the same direction.
From RSI to RSIHF: making every improvement respond to human needs
RSI Arena demonstrates the feasibility of automated post-training, but tests only one base model, runs each training agent only once, and does not recursively update the trainers themselves during the experiment. With these limitations, it does not establish reliable, sustained recursive self-improvement.
The paper proposes RSIHF (Recursive Self-Improvement with Human Feedback), a framework that incorporates human feedback into recursive self-improvement and brings user judgments into model evaluation and subsequent research and development. As experimental workflows become increasingly automated, humans still need to participate continuously in the relevant decisions, determining which problems are worth solving and what kinds of changes are worth pursuing.
The human contribution carries real weight in this process. Whether an answer is trustworthy, whether an act of assistance is useful, and who ultimately bears the consequences of an error are judgments rooted in people's real experiences, their actual circumstances and their hopes for life. It is human judgment that gives the standard of “better” a basis and connects otherwise abstract scoring metrics with our real lives.
Voting is only the starting point for understanding users' real needs. Under the RSIHF framework, researchers still need to evaluate model capabilities and safety independently, to determine in which situations models should seek human help, how to interpret the feedback collected, and how to keep calibrating the direction of optimization over multiple rounds of iteration. Human participation is precisely what makes the course of AI R&D open to scrutiny, explanation and even a different choice of direction.
Returning to the experiment's initial question, AI can indeed accelerate the cycle of model improvement and extend its reach, but only human experience and choices can determine where this path ultimately leads. As technical capabilities continue to grow, researchers must keep asking whose time these advances are saving, whose difficulties they are easing, and for whom they are creating new possibilities.
Statistically, every user vote is merely a record of preference. But on the other side of the screen is a real user with a specific problem.
The next round of model training can begin with that vote. But the technological progress we hope for must ultimately return to the everyday life of that real user.
This article comes from the Synced WeChat account. Author: Synced.
Related topics
After reading
Join the AINRK community
Read how to join the community and connect with people working in AI.




