AI guides

TestPrism evaluates agent-generated tests and exposes reliability gaps across implementations

By LRST1 viewsSource: 新智元
TestPrism evaluates generated tests across correct and incorrect implementations

The benchmark checks tests against correct alternatives and flawed implementations. The strongest tested configuration achieves 28% joint success; TestHelix improves results through cross-validation.

Coding agents are becoming stronger at writing code, and their test-writing abilities are improving too. From adding unit tests for functions and writing reproductions for GitHub issues to building regression tests for entire repositories, frontier coding agents such as Codex and Claude Code can already browse repositories, modify implementations and continually refine tests through execution feedback.

The role of tests is changing accordingly. Beyond safeguarding software quality, they increasingly provide the basis for agents' self-verification, evidence for code review and even reward signals in reinforcement learning.

The quality of the tests directly determines whether these downstream stages can trust their judgments.

So what makes a good test?

Mainstream test-generation benchmarks usually employ Single Reference Evaluation: a generated test counts as successful if it fails on the original code and passes on a Gold Solution.

The problem is that the same requirement often admits multiple correct implementations, as well as many seemingly reasonable implementations with hidden defects.

A test may happen to "recognize" the gold solution while letting genuinely buggy code pass. Or it may treat incidental implementation details of the gold solution as requirements, rejecting other fully correct approaches. Evaluating against only one reference can considerably overestimate test quality.

Recently, Jiaheng Liu's team at Nanjing University proposed TestPrism, a new test-generation benchmark with 300 test tasks from 17 sources and 3,000 candidate implementations. Each task has a verified panel of correct and incorrect implementations. Its core metric is Joint Success Function (JSF): a test must fail on the original code, accept every correct implementation and reject every incorrect implementation.

Across 14 coding-agent configurations, the strongest achieved only 28.00% JSF, despite a 59.67% success rate under single-reference evaluation. More than half of its tests that passed the reference-based check were no longer reliable against another set of implementations.

The researchers further had experts annotate 286 failed trajectories, identifying seven test-generation failure mechanisms in three broad categories. They also proposed TestHelix, which combines heterogeneous synthesis, peer cross-validation and recursive self-improvement to raise JSF by 11.67–12.00 percentage points under the same inference budget.

Article illustration

Paper: https://doi.org/10.48550/arXiv.2610.12289

Research data and code: https://github.com/TestPrism/TestPrism

What is missing from test evaluation?

Evaluating test generation is not a new problem.

Existing benchmarks cover different task granularities. TestGenEval, CLOVER and MultiFileTest focus on function-, file- or multifile-level test writing, while SWT Bench and TDD Bench Verified bring test generation into real repositories, requiring tests to reproduce problems described in GitHub issues.

They generally use the same success criterion: the test fails before a fix (Fail) and passes after the reference fix (Pass), known as Fail-to-Pass.

This criterion is simple and executable, but assumes that distinguishing "before the fix" from "after the reference fix" is sufficient to judge any implementation's correctness. In practice, that assumption is difficult to sustain.

On one hand, a test may omit key inputs or use weak assertions, so both the reference solution and defective implementations pass;

On the other, it may overdepend on the reference solution's specific approach, such as fixed internal function names, output formats or intermediate states, incorrectly rejecting other implementations that meet the same requirements.

Work such as OmniCode and SWE Mutation has begun introducing implementation variants, but mainly tests whether generated tests reject incorrect variants. Systematic evaluation of "accepting correct alternatives" remains missing.

Article illustration

Table 1 | Evaluation scope and objectives of existing test-generation benchmarks (Table 1 of the original paper).

The paper therefore poses a central question: How robust is the evaluation of generated tests to variation across implementations?

In other words, does an assessment of test quality still hold when implementations change? A reliable test should act like a prism separating implementations of the same requirement: tolerant of correct ones and sensitive to incorrect ones.

A panel of correct and incorrect implementations for every task

To answer this question, the researchers built TestPrism. Instead of providing only one reference solution, it gives each testing task a Candidate Panel: one reference implementation, four correct alternatives and five incorrect implementations. The panel is entirely hidden from agents during test generation. At evaluation, tests run separately on the original code and every implementation in the panel.

Article illustration

Figure 1 | TestPrism construction and evaluation pipeline (Fig. 1 of the original paper).

Construction has four steps.

Step one: task screening and adaptation

The researchers collected tasks from 17 public benchmarks, including the widely used SWE-bench and Terminal-Bench families, as well as SWE PolyBench, SWE rebench, EvoEval, USACO and AlgoTune. They cover repository maintenance, evolving function specifications, competitive programming and numerical optimization. Sources must have traceable test provenance and documented validation procedures.

Adapters then convert tasks from different sources into Harbor's task format while retaining original requirements, initial code states, validators, reference implementations, dependencies, services and execution restrictions.

Step two: task selection

The team screened tasks using three metrics: effective lines of code in the reference implementation (Solution Size), the number of test functions or checks (Test Count), and tests per modified file (Test Density).

Screening thresholds were calibrated to each source's own distribution, with source quotas balancing representation. TestPrism ultimately contains 2,000 tasks: 1,700 form a training set for TestHelix's later strategy optimization, and 300 form the test set. The training and test sets do not overlap.

Step three: generate and validate candidate implementations

Before generating candidates, the team rechecked each task's original validator: it must fail in the initial state and pass on the reference implementation.

Two experts with development experience independently assessed requirements and validators for reference overfitting and coverage gaps. They discussed disagreements to clarify them, removing tasks that remained doubtful (Cohen's κ = 0.68).

Candidates themselves come from different coding agents and models of different capabilities actually solving the tasks. Unlike conventional Mutation Testing, which injects defects artificially, this captures both genuine alternative solutions and errors agents naturally make while solving problems.

After removing empty submissions, exact duplicates and inapplicable patches, the team ran the original validators in clean environments. Implementations passing every check were labeled correct; those failing any check were labeled incorrect. These labels remain fixed in later evaluation.

Step four: diversity filtering and task conversion

If the four correct implementations were merely slight rewrites of the reference, the panel would lose its purpose.

The team therefore required the correct group to meet an average AST-distance criterion. Incorrect implementations count as diverse when their sets of failed checks differ; when those sets match, they must also pass an AST Jaccard-distance check. Panels are repeatedly generated and candidates replaced until both groups meet the diversity requirements.

Finally, a test-generation adapter turns each programming task into a test-writing contract: agents must use the original requirements to submit tests, fixtures and an executable entry script in the specified directory.

Data overview

The 300 tasks in TestPrism's test set cover six languages, including Python, JavaScript/TypeScript, Java, C/C++ and Go. They come from 67 repositories and projects and include 3,000 labeled candidate implementations.

Tasks fall into six broad categories and 34 subcategories: bug fixing (167), programs and algorithms (53), performance optimization (28), feature extension (20), projects and workflows (19), and refactoring and migration (13).

Article illustration

Figure 2 | TestPrism test-set task categories (Fig. 7 of the original paper).

Evaluation metrics

From one reference to joint success

On this basis, TestPrism defines complementary evaluation metrics:

  • Reference Acceptance Rate (RAR): the proportion of tasks whose tests fail on the original code and pass on the reference implementation—the traditional single-reference evaluation;
  • Valid Acceptance Rate (VAR): the average proportion of the four correct alternatives accepted by tests;
  • Invalid Rejection Rate (IRR): the average proportion of the five incorrect implementations rejected by tests;
  • H: the harmonic mean of VAR and IRR, measuring their balance;
  • Change Coverage and Entry Coverage: using the definitions from SWT Bench and TestExplora, respectively, these measure how much reference-modified code and relevant entry-point code tests cover.

The core Joint Success Function (JSF) requires three conditions simultaneously: fail on the original code, accept every correct implementation in the panel and reject every incorrect one. If the panel shrinks to the reference implementation alone, JSF reduces to RAR.

Single-reference evaluation is therefore JSF's most permissive special case. JSF can never exceed RAR. Their gap is precisely the proportion of tests that pass the reference solution but fail scrutiny against more implementations.

RQ1

Can frontier coding agents write reliable tests?

The first research question asks whether tests generated by current coding agents can consistently distinguish correct from incorrect implementations.

The researchers evaluated 14 coding-agent configurations from six model families. The main table reports each family's highest-JSF configuration, prioritizing each model's native coding agent to reflect the close coupling between model training and the agent framework.

Article illustration

Table 2 | Each model family's best configuration on TestPrism (Table 2 of the original paper).

Passing the reference does not make a test reliable

The results show that reliable test generation remains far from solved. The strongest, Claude Fable 5.1, achieved 59.67% success in single-reference evaluation but only 28.00% JSF. Kimi K3, Qwen3.8 Max and GPT 5.6 sol -max all had RAR between 54% and 57%, yet JSF of only 23%–25%;

DeepSeek V4 Pro and GLM 5.3 reached only 13.33% and 11.33% JSF. For every model, a substantial share of tests deemed successful under traditional evaluation failed when facing more implementations.

Accepting correct implementations is often harder than rejecting incorrect ones

Breaking down the results, five of the six configurations had IRR above VAR. This shows that tests written by current agents more often err by being too strict: treating reference-specific details as requirements and wrongly rejecting other valid implementations.

GLM 5.3 is a typical example: its IRR was 86.33%, the highest of all configurations, but VAR was only 50.33%, and its JSF was the lowest.

Qwen3.8 Max showed the reverse: the highest VAR (77.00%) but only 66.00% IRR. Being good only at rejecting incorrect implementations is therefore insufficient. Reliable tests must balance tolerance and sensitivity. Across all 14 configurations, the harmonic mean H of VAR and IRR was also positively correlated with JSF.

Coverage matters, but it is not everything

Coverage was also strongly correlated with JSF. Across all 14 configurations, the Pearson correlations of Change Coverage and Entry Coverage with JSF were 0.91 and 0.96, respectively. But reaching relevant code is only a prerequisite for judging it correctly: executing a piece of logic does not mean the assertions can correctly assess its behavior.

RQ1's central finding is that frontier models remain far from reliable test generation. Accepting correct implementations and rejecting incorrect ones reveal complementary weaknesses, and their balance, together with coverage, affects joint success.

RQ2

Do larger, more diverse candidate panels make it harder for tests to slip through?

The second question concerns the candidate panel itself: what happens to test success as more and more diverse implementations are gradually added?

The researchers randomly sampled 50 tasks and used the same construction procedure to expand each panel from 10 implementations to 15: one reference, seven additional correct implementations and seven additional incorrect ones.

Keeping tests unchanged, they began with a reference-only panel and added implementations in three ways: correct ones only, incorrect ones only, and alternating between the two.

Article illustration

Figure 3 | Changes in JSF and candidate diversity under three implementation-addition methods (Fig. 3 of the original paper).

Average JSF declined continuously from 49.33% with only the reference: it reached 38.10% after adding five correct implementations, 30.49% after adding five incorrect ones and 24.33% after nine alternating additions. Once each sequence had added all its implementations, the three endpoints were 37.33%, 29.33% and 23.33%, respectively.

The tests did not change at all. Only the panel's demands changed: more correct implementations test whether tests are too strict; more incorrect implementations test whether they are too permissive.

JSF is thus not a static score but a dynamic reliability measure that changes with the panel. Which implementations it contains matters as much as how many. Single-reference evaluation, the special case with one implementation, is the weakest check and an upper bound on expanded-panel success. RQ2's central finding is that diverse candidates expose test errors small panels cannot detect.

RQ3

Where exactly do agent-generated tests go wrong?

The first two questions reveal how unreliable tests are, but not why.

The researchers therefore sampled 50 failed trajectories with JSF zero from each of the six model families' best configurations. After excluding 14 timeout trajectories, two experts independently annotated all 286 (Cohen's κ = 0.81). Disagreements brought in a third expert for joint discussion and resolution.

Experts inspected task requirements, submitted tests, execution results on original code and the reference implementation, results on every candidate, and corresponding patch evidence. They identified seven specific failure mechanisms in three broad categories.

Article illustration

Figure 4 | Metric profiles by model (a) and failure-mechanism distribution across 286 failed trajectories (b) (Fig. 4 of the original paper).

Coverage gaps: reaching a feature but missing key inputs

Coverage Gaps accounted for 34.6% of failures: Missing Inputs (30.8%) and Untested Requirements (3.8%). The former was among the most common individual errors: a test executed an operation but did not cover input combinations that expose the defect.

For example, in a poker-hand classification task, generated tests never compared an ace-low straight with three of a kind. An incorrect implementation that classified that straight as "high card" therefore passed.

Faulty checks: executing the behavior but judging it incorrectly

Faulty Checks were the largest category at 50.0%, comprising Weak Checks (7.7%), Wrong Criteria (26.9%) and Overfitting (15.4%).

This category shows that reaching relevant behavior does not guarantee judging it correctly. Overfitting is precisely the excessive strictness observed in RQ1: treating incidental reference details as the requirements themselves.

For example, in a bracket-parsing task, a test did construct input containing mismatched brackets, but checked only the number of output results, not their contents. It therefore accepted an incorrect implementation that discarded valid groups.

Notably, the tests in both the poker and bracket examples failed on the original code and passed on the reference. Both would be judged successful in single-reference evaluation. Only additional incorrect implementations exposed the missing input combinations and weak assertions.

Execution logic errors: the test itself fails to run correctly

Execution Logic Errors accounted for 15.4%: fixture and API misuse (7.7%), and command-execution and output-parsing errors (7.7%).

In an EvoEval task, for example, a test stored the function under test on the test class and called it through self, inadvertently binding an extra argument. The test consequently failed even on the reference implementation. Such problems directly reduce overall test reliability, but unlike the first two categories, single-reference evaluation also detects them. They are not its "blind spot."

RQ3's central finding is that failures arise from missing relevant behavior, misjudging implementation correctness or failure of the test's own execution. Missing input combinations and weak assertions can survive reference-based checks and emerge only against more incorrect implementations.

RQ4

How can tests generalize across implementations?

The researchers proposed TestHelix to address these problems. Its starting point is direct: if checking against one implementation cannot uncover test problems, actively construct multiple implementations during generation and subject tests to multiple checks.

Article illustration

Figure 5 | The TestHelix framework (Fig. 2 of the original paper).

Heterogeneous synthesis: different agents each produce tests and a fix

TestHelix assigns different exploration strategies to multiple isolated coding agents. Using only public task requirements, each independently writes a test T and a fix P. Every pair must be internally consistent: the test must reproducibly fail on the original code and pass on its own fix. Agents keep revising through local execution feedback until these conditions hold or the budget runs out.

Peer cross-validation: check your tests with others' fixes

Passing only on one's own fix can still be a self-consistent mistake. TestHelix therefore cross-executes every test on other agents' fixes, exposing excessively strict tests or incomplete fixes.

Meanwhile, an independent review agent that sees no fix code examines assertions one by one against public requirements. Every blocking comment must be grounded in those public requirements.

Review-confirmed counterexamples are used to revise both tests and fixes. Once several different qualifying fixes pass accepted tests, the system also borrows from differential testing to actively seek inputs that distinguish them. Finally, compatible tests are merged and verified again in a fresh environment.

Recursive self-improvement: learn better test-generation strategies from training tasks

Starting from a base strategy, TestHelix uses AutoSaddler for reflective evolution: select a parent strategy, diagnose its execution trajectories, propose revisions, compare parent and child on the same tasks, and retain reusable lessons for subsequent rounds.

Optimization uses 600 tasks from TestPrism's training set, completely isolated from the test set, and never accesses hidden candidate panels. Notably, the optimization objective is the denser RAR signal rather than stricter JSF, since the latter often struggles to reward incremental strategy improvements. Heterogeneous synthesis and peer cross-validation instead carry the requirement to distinguish multiple implementations.

After optimization, the strategy is frozen and applied directly to GPT 5.6 sol and Kimi K3.

Experimental results

The researchers compared five TestHelix configurations on GPT 5.6 sol and Kimi K3, using their native coding agents, Codex and Kimi Code respectively, as baselines. Within each model group, all configurations use the same base model and tools and share the same per-task inference-budget ceiling.

Article illustration

Table 3 | TestHelix ablations and comparisons with native coding agents (Table 3 of the original paper).

Generating three test-and-fix pairs and randomly submitting one improved at most one percentage point over generating a single pair. With the same three pairs, enabling full peer cross-validation increased JSF by more than six percentage points: GPT 5.6 sol rose from 21.00% to 27.67%, and Kimi K3 from 22.67% to 29.00%. Increasing the pair count from three to five added only 0.67 percentage points. Introducing recursive self-improvement into the three-pair configuration added a further 4.33–4.67 percentage points.

The final configuration achieved 32.33% and 33.33% JSF on the two models, respectively: gains of 11.67–12.00 percentage points over single-pair generation and 8.67–9.00 percentage points over their native coding agents. Under a fixed inference budget, improving "how tests are generated and checked" brought clearly greater benefits than simply allocating the budget to more agents.

RQ4's central finding is that heterogeneous synthesis helps tests generalize across implementations. Peer cross-validation uses diverse fixes to examine and correct tests, while recursive self-improvement accumulates test-generation strategies across tasks. Both contributed clear gains in ablation experiments.

What comes next?

As coding agents increasingly rely on tests to verify their own work, standards for judging test quality must rise too.

This study shows that single-reference evaluation systematically overestimates test quality. Distinguishing the pre-fix and post-fix states does not mean a test can correctly judge other implementations of the same requirement. Even the strongest evaluated frontier models achieve below 30% joint success against multiple correct and incorrect implementations.

The study also reveals the two sides of the problem. Tests may be too permissive, omitting key inputs or using superficial checks that let defective implementations pass. Or they may be too strict, treating incidental reference details as requirements and wrongly rejecting valid implementations.

The former can hide in single-reference evaluation; the latter cannot be discovered there at all. True test reliability becomes clear only when evaluation examines both acceptance and rejection.

From an evaluation perspective, TestPrism offers a more realistic check: evaluate tests against verified, diverse implementations rather than relying on one gold solution.

From a methodological perspective, TestHelix shows that improving test reliability does not necessarily require stronger base models. Having agents provide counterexamples for each other, cross-check conclusions and consolidate useful experience into strategies can also deliver substantial benefits.

The paper also notes limitations: each TestPrism task contains only 10 candidates; equal numbers of correct and incorrect implementations do not represent the real distribution; and AST-based diversity metrics cannot guarantee complete semantic coverage. Extending evaluation to evolving requirements, broader implementation spaces and more execution environments remains a direction for future exploration.

References:

https://doi.org/10.48550/arXiv.2610.12289

This article comes from "Xinzhiyuan." Author: "LRST."

Related topics

After reading

Join the AINRK community

Read how to join the community and connect with people working in AI.

Community information (opens in a new tab)