AI guides

NUS Introduces SO-Arena: Measuring the Breadth and Reliability of System One Decisions

By 新智元0 viewsSource: 新智元
SO-Arena evaluates System One models across capabilities, option-order stability, and absent correct answers.

NUS xML Lab's SO-Arena uses 2,919 tasks to evaluate capability, option-order stability, and recognition of absent correct answers. Capability and reliability rankings differ, while local H100 and remote API latency measurements use different conditions.

Editorial verification (November 22, 2026): The SO-Arena paper (opens in a new tab)uses different model sets and task selections for capability, stability, and reliability; denominators and rankings must not be mixed. Reliability R specifically refers to two operational measures—stability under option reordering and recognition that the correct option is absent—not certification of all aspects of production reliability. In the latency comparison, open-source models run one task at a time locally on a single H100, while closed-source APIs include network and provider queuing time. This is not an experiment under identical deployment conditions that changes only whether the model thinks; the entire speed difference cannot be attributed to System One versus System Two.

When a model makes a decision in a single forward pass, how should we judge whether it is both correct and dependable?

Routing user requests, choosing the next tool action, and checking whether a task is complete—many routine decisions in software workflows do not require a model to first write a long reasoning trace.

System One models are designed for precisely this: they output a probability distribution over candidate outcomes in a single forward pass, with low latency and structured output that downstream programs can use directly.

But does a high score on a leaderboard mean a model is useful? If the reference answer itself is disputed, can the score still be trusted? If all questions have similar difficulty, can strong and weak models still be distinguished? If a test assesses only single-step judgments, can the model complete an entire task in a web or tool environment?

Going further: if the same options are merely reordered, should the model's answer change with them? If the correct option is not in the list at all, can the model recognize that "there is no suitable answer here"?

The latest work from the xML Lab team at the National University of Singapore, with Lichen Li as first author and Xinchao Wang as corresponding author, begins with these questions and builds a benchmark that characterizes both the range of System One capabilities and the reliability of their decisions.

Article illustration

Paper: https://arxiv.org/html/2611.15777v1

Project page: https://lichenlillc.github.io/SO-Arena

Hugging Face: https://huggingface.co/SO-Arena

GitHub: https://github.com/holdrain/SO-Arena

Article illustration

Figure 1 | SO-Arena's four design elements: verifying reference labels; stratifying difficulty to distinguish models; evaluating long-horizon interaction in sandboxes built for System One; and testing reliability when options are reordered or the correct one is absent. Figure 1 in the original paper.

Evaluating System One Models

What Questions Need to Be Answered?

SO-Arena looks beyond the overall score at four requirements a benchmark should meet to characterize System One models meaningfully.

Trustworthy Labels | Reference Answers Must Be Supported by Evidence the Model Can See

A five-star review saying "Bought this for my parents" does not itself explain why the rating is five stars. In contrast, a one-star review saying "Counterfeit, do not buy!" has evidence consistent with its label. If the reference answer cannot be derived from the input visible to the model, an "incorrect" response may not be the model's fault. SO-Arena draws questions from traceable sources, manually verifies that every reference label is supported by the input information, and strictly separates evaluation references from model inputs.

Difficulty Stratification | Hard Questions Reveal the Differences

If every question is equally easy, three models might all answer correctly, leaving scores unable to distinguish them. In nine of its 11 domains, SO-Arena defines difficulty levels according to domain-specific criteria. Mathematics uses the number of computation steps and source difficulty, while task-completion verification distinguishes judgments based on "directly inspecting the current state," "combining multiple pieces of evidence," and "relying on execution history."

Interactive Evaluation | More Than Judging One Step: Completing the Task

In real applications, System One models often need to make many consecutive decisions: which element to click, what value to enter, and which tool to call. If the earlier step is wrong, the next may be impossible. SO-Arena lets models act step by step in web and tool sandboxes. Each step's execution result becomes the next input, and the whole task is scored using the original benchmark's success criteria.

Decision Reliability | Does the Judgment Hold When Options Are Reordered or the Correct Answer Is Removed?

Reordering A, B, C as C, A, B does not change the options' content, so the model should still select the same answer. Removing the correct option should increase its probability for "no suitable answer." SO-Arena combines these two checks into a separate reliability score, reported alongside capability.

Capability breadth and decision reliability are two complementary dimensions for characterizing System One models.

How Is SO-Arena Constructed?

SO-Arena unifies static judgments and environmental interaction through the same decision interface: given information, a decision objective, and candidate outcomes, the model returns probabilities for each candidate. The entire benchmark contains 2,919 tasks, scored at the original-task level.

Article illustration

Figure 2 | SO-Arena overview. (1) Tasks are drawn or derived from 40 public sources, with human and LLM review removing questions with questionable labels. (2) The 2,919 tasks cover five capability families and 11 domains, nine stratified by difficulty, with static and environment-based questions. (3) Capability C uses each domain's hardest level and weights domains equally; reliability R combines option reordering and absence of the correct option. Figure 2 in the original paper.

Traceable Sources, Item-by-Item Verification

Tasks come from 40 public datasets, benchmarks, and environments, including GoEmotions for affect understanding, MATH500 and AIME for mathematics, GPQA Diamond for scientific reasoning, Mind2Web for web interaction, and BFCL and ToolSandbox for tool use.

The researchers retain original labels as references, then use human and LLM review to remove questions with questionable labels. Each question records its source, version, and original identifier; model-generated reference labels are also identified in the metadata.

Five Capability Families, Eleven Domains

The 11 domains are grouped into five capability families: semantic understanding, knowledge and reasoning, safety and policy compliance, evidence evaluation, and agentic task completion. They cover typical decision scenarios ranging from understanding a sentence to completing an entire task in an environment. Task counts and difficulty distributions for each domain appear below.

Model outputs are unified into three decision formats: choice selects among explicitly provided candidates; noul answers a yes/no question and gives the affirmative probability; score assigns a rating on an ordered scale while retaining the source's native scale.

Article illustration

Figure 3 | Benchmark composition. Left: capability families in the inner ring and domains in the outer ring. Right: task distributions across difficulty levels for each domain, with bar labels showing task counts. Difficulty levels are defined within domains and are not calibrated across domains. Figure 3 in the original paper.

Static and Environment Tasks

The 2,300 static tasks ask models to judge fixed inputs, while 619 environment tasks put them in web or tool environments. The 337 web tasks include 100 single-step judgments and 237 tasks requiring consecutive decisions along a reference path. The 282 tool-calling episodes use BFCL or ToolSandbox simulators to execute the model's selected actions step by step and return new observations.

For task-completion verification, SO-Arena also constructs execution records with "the same endpoint, different processes": two trajectories ultimately return an item to the required location, but in one, the item was in the wrong place while an electrical appliance was operating. The final state alone cannot distinguish them; the model must inspect execution history.

Two Scores: Capability C and Reliability R

Capability C takes the hardest level in each domain—both hard and very hard for mathematics, and all tasks for domains without levels—and then averages the 11 domains equally. This prevents larger domains from dominating the ranking and makes the leaderboard more sensitive to difficult tasks.

Reliability R has two components.

Order stability S: each question is evaluated with 12 additional permutations beyond the original ordering. Answers are compared by option content rather than position, with agreement attributable to chance subtracted;

Recognition of an absent correct answer N: each question appears twice, once complete and once with the correct option removed. The model's probability for "no correct answer" is used to distinguish the two, with N = 2·AUC−1. The two components are combined through their geometric mean; R is positive only when both are above chance.

Experimental Results

The team evaluated 21 models: 16 open-source System One models ranging from 0.4B to 31B; StartLux Auto, which can switch automatically between System One and System Two; two closed-source decision APIs, Jev API 1.14 and OpenAI Decisions; and two closed-source System Two reference models, GPT-6 Sol and GPT-6 Luna. All models were run with the settings specified in their developers' documentation.

Article illustration

Figure 4 | SO-Arena at a glance. Left: leaderboard scores for 21 models, with darker colors indicating higher scores and horizontal grouping by scale and openness. Right: capability and option-order stability for the 19 models with stability measurements; the red line marks the Pareto frontier. Figure 4 in the original paper.

Who Is Strongest? System Two Still Leads, but the Gap Varies by Domain

On overall scores, GPT-6 Sol at 82.3 and GPT-6 Luna at 79.7 still lead all System One models. The strongest System One models, GLiDE and KnowLine-31B, both score 70.2, about 10 points behind Luna.

The right-hand plot combines capability and option-order stability. Its Pareto frontier consists only of GLiDE and Jeva-4B, the former highest in capability and the latter highest in stability.

The trade-off is speed. Every open-source System One model processes a task in no more than 0.3 seconds on a single H100, with GLiDE averaging approximately 113 milliseconds. GPT-6 Sol and Luna have end-to-end times close to three seconds, a gap of several tens of times. StartLux Auto, at around three seconds on the right of the chart, switches into thinking mode as needed and is therefore not included in that category.

Article illustration

Figure 5 | Capability and latency. The vertical axis shows leaderboard scores; the horizontal axis shows mean per-task latency on a logarithmic scale. The Pareto frontier comprises the best-performing System One models at each latency level. Open-source models run tasks sequentially on a single H100, whereas closed-source API times are end to end, including networking. Figure 5 in the original paper.

Where Are the Gaps? Three Groups of Domains, Three Answers

Using GPT-6 Luna as a reference and comparing the strongest open-source System One models at two scales—StartLux Instant at 4B and GLiDE at 27B—the 11 domains naturally fall into three groups:

Direct semantic judgments (knowledge and common sense, sentiment and emotion, retrieval and reranking): System One models have the smallest gaps from Luna. GLiDE exceeds it by 5.0 and 4.8 points on knowledge and common sense and sentiment and emotion, respectively, while trailing by 5.8 points on retrieval and reranking. Neither a stronger backbone nor additional thinking brings large gains.

Fine-grained, grounded decisions (intent routing, safety, task-completion verification, tool calling, and web interaction): GLiDE's gaps from Luna are all within four points, while StartLux Instant falls as much as 29 points behind, on web interaction. Within the same model family, switching to a larger, newer backbone delivers much greater average gains than thinking: +25.6 points on tool calling and +23.9 on web interaction, compared with only about two to three points from thinking in those domains.

Multistep derivation (mathematics, scientific reasoning, and code comprehension): both System One models lag substantially, with GLiDE 26.7 points behind in mathematics. Thinking helps most here: in mathematics, StartLux Auto's score rises from 50.8 to 92.5 when thinking is enabled, approaching Luna.

Article illustration

Figure 6 | The gap from System Two and how it narrows vary by domain. (a) Score differences between two System One models and GPT-6 Luna at medium and hard levels, including 95% confidence intervals. (b) Gains within the same family from a small model to a 27B model. (c) Gains from StartLux Auto always thinking versus never thinking. Figure 6 in the original paper.

Are the Decisions Reliable? Capability and Reliability Rankings Differ

Reliability measurements for 18 System One models show that strong capability does not automatically mean reliability. JevAny-27B, the most reliable model with R = 0.84, ranks only 11th for capability. GLiDE, the strongest System One model in capability with R = 0.78, ranks only seventh for reliability.

The two components also behave very differently. Fourteen of the 18 models have order stability S above 0.8, and most retain their original answer when options merely change order. Recognition of an absent correct answer N ranges from near chance to approximately 0.8, and model size alone does not determine it: some 4B models have higher N than certain 9B and 27B models.

Article illustration

Figure 7 | Reliability and its two components for 18 System One models. (a) Reliability R, with each model's leaderboard score labeled to the left of its bar. (b) Absent-answer recognition N and order stability S; zero represents chance, and dashed lines indicate equal-R curves. Figure 7 in the original paper.

Why? Stability Follows Capability, but Absent-Answer Recognition Needs Its Own Measurement

Order stability broadly improves with capability, with most models close to the trend line. Moreover, only 4.3% of originally correct answers change after reordering, compared with 25.7% of originally incorrect answers. Instability occurs mainly where the model was already weak.

The relationship between absent-answer recognition and capability is much looser: at similar capability levels, N varies much more than S. Kev-27B, Jev API 1.14, and JevAny-27B all have higher N than the four strongest System One models by capability. A capability score can indicate whether a model is stable, but says little about whether it can notice that "the correct answer is not here."

Article illustration

Figure 8 | Order stability rises with capability, while absent-answer recognition follows it only loosely. (a) S against capability. (b) The proportions of originally correct and incorrect answers that change after reordering. (c) N against capability. Each point is a System One model, and the curves are quadratic trends. Figure 8 in the original paper.

These results make "reliable decision-making" more specific: maintaining a judgment when options change position, and noticing when the correct answer is absent. The latter cannot be read off accuracy and must be measured directly.

Conclusion

With traceable labels, stratified difficulty, interaction in sandboxes, and independent reliability scoring, SO-Arena provides a more complete "checkup" for System One models. Its core can be summarized as:

Assess capabilities by breadth and difficulty, and decisions by stability and awareness of their own limits.

Capability and reliability, measured side by side.

For developers incorporating System One models into real workflows, SO-Arena offers more than a leaderboard. It provides an evaluation perspective: alongside asking whether a model can answer correctly, ask whether it maintains its judgment when ordering changes and can honestly say "none apply" when no suitable option exists.

References:

https://arxiv.org/html/2611.15777v1

This article originally appeared on the WeChat account "Xinzhiyuan," written by Xinzhiyuan.

Related topics

After reading

Join the AINRK community

Read how to join the community and connect with people working in AI.

Community information (opens in a new tab)