AI guides

SPU benchmarks wet-lab rankings on 116 tasks; more reasoning does not always improve predictions

By 量子位0 viewsSource: 量子位
SPU evaluates wet-lab ranking predictions across 116 tasks and compares reasoning effort and performance

Across public papers and unpublished experiments, SPU compares 12 models in 13 configurations. Performance varies sharply by task, while higher API costs do not yield corresponding gains.

Wait, what? Have benchmarks really evolved this far already???

No more solving test questions this time. Instead, give AI a set of experimental conditions and let it do its own research and predict the results of real wet-lab experiments in advance……

For example, if the same reaction is run under several different conditions, which will yield more of the desired product? Among a set of antibody variants, which will have higher expression?

Keep the answers hidden first. Once AI has handed in its predictions, compare them with the measured results.

And you know what? This really is worth testing now.

Over the past two years, approaches to AI for Science have become increasingly sophisticated. AI can already search the literature and use software autonomously, and working continuously for hours to complete an analysis task is no problem either.

Naturally, people are moving on to the next stage: what kinds of scientific research capabilities do we actually want AI to possess?

Predicting wet-lab results is one capability with considerable practical value.

It requires AI to understand the experimental objective and uncover relevant mechanisms in existing knowledge, so it can better help researchers select research directions worth exploring further.

The new benchmark led by UniPat AI, Science Prediction Universe (SPU for short), has keenly recognized this point.

Article illustration

There are 116 tasks and 4,207 wet-lab experiments covering chemistry, materials science, biology and medicine. The benchmark's advisers and wet-lab collaborators also come from universities such as Stanford, Cornell and Harvard, as well as frontier research institutions and biotechnology companies.

Models work an eight-hour shift for each run, equipped with one H20 GPU, and independently decide how to use retrieval and computing resources to produce experimental predictions.

The results took everyone by surprise.

Even models as strong as Opus 5.5 and GPT-6 Astra are, according to the data, still a long way from truly being able to predict experimental outcomes. . .

Article illustration

△ Average score of each configuration across all scored tasks

SPU has certainly given every scientific research model a tough challenge to chew on.

Scientific research capabilities that were once invisible and intangible have now become numerical values that can be evaluated consistently and studied on an ongoing basis.

There is now a more concrete basis for discussing how AI's scientific judgment develops step by step.

Wet-lab prediction: worth a separate exam for models

Returning to wet-lab prediction, the first point is that this is certainly not a new problem.

There have been many such moments in the history of science: a judgment is made before the experiment and then handed over to the experimental process to decide. If it proves correct, it is retained; if it is wrong, it is overturned.

The specific logic behind these judgments constitutes a scientist's taste.

With good enough taste and accurate enough intuition, scientists can achieve major results faster while saving valuable experimental resources.

After all, even a single wet-lab experiment requires chemists to prepare reagents, set reaction conditions and then separate the products. Biologists also have to go through lengthy protein expression… The time costs and equipment expenses alone are substantial, and if the results are unsatisfactory, several rounds have to be repeated.

So predicting before doing the experiment is the most valuable step in the research workflow.

The difference is that this used to rely entirely on scientists themselves. Now, in the AI era, models also need this ability, along with an exam paper that can score it.

Existing benchmarks do not yet cover this fully: BrainBench, for example, asks models to judge experimental outcomes directly. ScienceAgentBench focuses on completing analyses using existing data. Long-horizon task benchmarks such as Terminal-Bench 4.0 and Long-Horizon-TB mainly use predefined tests to check task outputs.

SPU differs from all of these.

It places greater emphasis on evaluating AI's autonomous research process together with its predictions of wet-lab outcomes, covering prediction performance while also testing scientific-tool use and the execution of long-horizon tasks.

Article illustration

△ Chart generated by AI

SPU also organizes this kind of judgment into clearly specified tasks.

The 100 public tasks come from 85 recently published papers. Domain experts extract candidates, materials, conditions and measurement methods from the studies, retaining the complete quantitative results as the basis for scoring.

Another 16 private tasks come from four sets of unpublished collaborative wet-lab experiments involving VHH nanobodies, reaction conditions, electrolytes and Magrolimab variants.

In total, there are 32 subfields and 4,207 experiments. Published studies provide a cross-disciplinary source of tasks, with measured results hidden from the models, while unpublished experiments add a test independent of the answers in public papers.

To some extent, this can prevent models from having seen the questions in advance.

Article illustration

△ SPU task coverage and sources

The task-selection process is also carefully considered.

Experts prioritize studies with complete measurements and clearly specified conditions. Simple problems that can be ranked by following a single trend are excluded outright, while problems with numerical values that are too close together are also considered to have limited value as a reference.

The questions that ultimately remain are harder and better at distinguishing performance.

For the prediction process, models are given eight hours per run, with an eight-core CPU, 16 GiB of memory and one H20 GPU, and are allowed to search online and install scientific software.

During this time, models independently gather evidence and choose computational methods, then submit for each comparison group a predicted ranking from lowest to highest.

To prevent contamination from memorized information, scoring also takes place entirely in separate containers disconnected from the internet. AI is prohibited from retrieving the original study results, and tool outputs are automatically scanned to identify identifiers such as the source papers' DOIs and titles.

Finally, the system uses Spearman's correlation coefficient to compare the predicted and experimentally measured rankings.

The closer the coefficient is to 1, the more closely the ranking matches the experimental results.

Article illustration

△ SPU evaluation workflow

Interestingly, SPU also considers many details that other benchmarks have not taken into account.

For example, when it comes to the final score, SPU considers whether it might just be down to luck. What if it was simply a lucky fluke?

It therefore generates another 4,000 random rankings for each question and uses the 95th percentile as the chance-level threshold, checking whether a model's submission clearly exceeds this reference. Different tasks also have their corresponding reference criteria.

The design of SPU as a whole can be described as scientifically rigorous.

Within it, models have to decide for themselves, step by step, what is worth computing and hand in their final predictions.

This design helps measure current AI capabilities in predicting wet-lab outcomes, while also pointing the way for research teams to keep improving experiments using unified metrics.

That is the idea. Next, let us look together at the report cards of the first batch of frontier models. Some of the details are worth singling out for a separate discussion.

A new test makes frontier models crack under pressure

SPU's first systematic evaluation covers 12 frontier models and 13 configurations. A configuration here refers to the model, its reasoning effort and the framework used for the model.

First, the overall results.

In a word: none is very good.

Across all scored tasks, Claude Opus 5.5 achieved the highest average score, at 0.486. Even so, it exceeded chance level on only half the tasks.

This was much the same across the 13 configurations: the proportion of tasks exceeding the chance-level threshold was only 34%~52%.

In other words, every model here still has a very long way to go before it can make truly reliable wet-lab predictions.

Article illustration

△ Comparative analysis of behavior in the leading and trailing groups

There are nevertheless differences between the models, and average performance can be divided into two tiers:

the leading group (six models) and the trailing group (seven models).

On the same tasks, the leading group's average exceeds that of the other seven by roughly 0.15 points.

Another detail is worth noting: the leading group scores below 0 on around 9% of tasks, while this proportion reaches as high as 22% for the other seven configurations.

Here, 0 is the average score of valid random rankings. A negative score means that the predicted and actual rankings are negatively correlated.

Where exactly does the gap come from? The research team dug a little deeper.

They first combine the predictions of the other models on the same task into a reference ranking, then form pairs of experiments and compare them with the measured results to check whether each individual model has ordered them correctly.

For experimental pairs ordered correctly by the reference ranking, the leading group gets 86% correct, compared with 78% for the other seven configurations. For pairs ordered incorrectly by the reference ranking, the proportion independently ordered correctly is around 30% in both groups.

So part of the leading group's advantage lies precisely in cases where the shared prediction is correct: the leading group is consistently more reliable in those cases.

As for why models deviate from correct judgments, the run logs provide more concrete clues.

Take the solution task as an example. The same polymerization reaction is run with 12 different solvents in sequence, and the task is to rank them by the mass of polymer finally isolated.

Opus 5.5 scores 0.874, while Qwen3.8 scores -0.126. Pairing the 12 candidate solvent conditions produces 66 pairs in total. The former orders 56 correctly, and the latter orders 30 correctly.

Article illustration

△ Two runs on the same task: Opus and Qwen

The analysis found that this was because the two had focused on different factors from the outset:

Qwen3.8 mainly ranked solvents by how much ethylene they could dissolve. Opus 5.5 paid more attention to the miscibility of the starting materials and the compatibility of the catalyst with the solvent.

Although Qwen3.8 also discovered this issue during its calculations, that ultimately did not directly change the basis the model used for its final ranking.

The stage at which the prediction was most likely to go wrong was therefore choosing the basis for judgment. The model also needs to learn to adjust its ranking approach immediately in response to clues, so these discoveries can actually influence the answer.

Looking at the breakdown, these differences also vary with the specific discipline and task.

Among the scored results for each discipline in the public tasks, Opus 5.5 has the highest average in chemistry, Fable 5.1 in materials science, GPT-6 Astra Medium in biology, and Opus 5 in medicine.

P.S. There were instances of refusal to answer by Fable 5.1 and Opus 5.5.

Article illustration

△ Scores of the models across the four disciplines

So if you plan to involve AI in a particular kind of research, consider first finding a relevant task and testing it in SPU. Confirm that the model really is suitable before proceeding with the actual experiment.

The private-task results make this point more concrete.

Twelve of the 13 configurations have average scores significantly above random ranking, with no clear differences detected between configurations. By contrast, the variation between tasks is especially striking.

For the same set of electrolytes, the ranking correlation coefficients of the 13 configurations cluster between 0.89 and 0.95 when predicting ionic conductivity. Switch to the electrochemical window, and the scores of the 12 scored configurations range from -0.55 to 0.07, with none exceeding the chance-level threshold.

The models have not changed, but changing the measurement target noticeably changes the distribution of scores.

So the next time a model is evaluated, saying only that it is good at chemistry is not specific enough. One also needs to look at the exact measurement target and experimental conditions it faces.

Article illustration

△ Prediction performance of the models on the 16 private tasks

Here is something else rather counterintuitive:

Generally speaking, giving a model more time and higher reasoning effort should keep improving its performance, right?

That is what people generally think, and that is what they generally do.

And guess what happened? The SPU results came out and delivered a slap in the face to that assumption—

Under the same eight-hour compute budget, the research team increased GPT-6 Astra's reasoning effort from medium to maximum. At maximum effort, median runtime rose to 4.8 times the previous level, and median API cost rose to 5.3 times the previous level.

But the prediction scores… barely moved.

Hmm? The research team even went back specifically to look through the model's run logs.

And here was the surprise: on the 79 completed tasks, GPT-6 Astra at maximum reasoning effort had typically already produced an answer when just 26% of its total runtime had elapsed (median).

The model did not sit idle afterward. It kept calculating and revising, but these were all minor adjustments that did not produce a difference in the results.

In other words, higher reasoning effort does not necessarily make the result better.

Article illustration

△ Resource input and performance of GPT-6 Astra at two reasoning-effort levels

This finding also offers a warning about model resource allocation. If more computation is to be added, do not rush in blindly: it takes time and effort without necessarily paying off.

Instead, one can inspect the run logs more closely, first fully understand which judgment the new evidence actually bears on, then selectively increase compute where it serves a specific purpose.

In fact, tool use also shows a similar phenomenon.

The median number of method categories used per run is four for GPT-6 Astra Max and one for Medium. Yet no statistically significant difference in scores is detected between the two configurations.

The reason is much the same: good tools were not being used where they mattered most.

Take Monte Carlo sampling: although the model spent most of its time calling this tool, it only added an uncertainty range to the final ranking and did not directly affect the result itself.

Article illustration

△ Scientific computing tool calls

So however many tools are used, it matters more that they are used precisely where needed.

GPU usage also offers an interesting perspective.

Despite ample GPU computing power being provided, GPU computations were successfully run only 255 times, fewer than one-fifth.

GPU usage also varies across models and disciplines. Simply counting the number of GPU calls therefore says little, on its own, about a model's scientific research capabilities.

To judge whether compute has helped, one must also look at what it was used to calculate, whether the computational results helped the prediction, and how those results influenced the final ranking.

The key is still to direct compute toward ranking-relevant physical quantities so models can make more accurate predictions.

Article illustration

△ GPU usage statistics

Looking at costs alongside this, the average API cost per run ranges from $0.22 to $65.91 across configurations, a full 300-fold difference.

This means SPU provides both prediction performance and model-call costs, offering a useful reference for teams preparing to deploy scientific research models.

From overall scores to traces on the same tasks, SPU unpacks model differences down to every detail of scientific judgment, with computing power as one example and cost as another.

How much do the values calculated by AI actually help prediction? When a prediction is wrong, which judgments should be revisited? These questions can all be analyzed further using the experimental results and run logs.

The next round of improvements then has concrete directions to try:

Using SPU, models can identify mistaken assumptions more promptly and retain or revise rankings on a firmer basis. After the changes, they can be put back on the same tasks to see whether their predictions are more accurate.

Bringing “testing understanding” into the exam room for models

One could say that SPU aligns closely with the patterns of AI4S development.

Evaluations of AI4S are gradually moving deeper into real experimental work, and benchmarks focused on experimental execution have also been emerging throughout this year.

But beyond execution, prediction, an unavoidable step in the research workflow and also the hardest to quantify, finally has a way to be measured.

The benefits of this are apparent in two ways:

Researchers can now analyze which measurement targets are easier to predict, which tasks differ the most, and how a mistaken judgment takes shape. This can both save expensive and valuable experimental resources and shorten decision-making time.

For model development, this also provides a more concrete reference for improvement.

As the range of experiment types expands, evaluations can answer more specific questions: for which materials, experimental conditions and measurement methods are AI's rankings more reliable and more useful as a reference?

Scientists can thereby continue to accumulate progress on the same kinds of problems.

Article illustration

In fact, SPU is only the tip of the iceberg in UniPat AI's series of real-work evaluations:

  • ScienceVision: focuses on how models obtain evidence from three-dimensional scientific objects and answer questions;
  • PaperBenchX: turns published research into executable, verifiable paper-replication tasks;
  • SaaS-Bench and EvoCode-Bench: examine real software workflows and development work under continually changing requirements, respectively. …

These efforts cover scientific research, office work and software development. Their common concern is how AI completes tasks once it enters specific work settings, and how the results are verified.

This also indirectly reflects UniPat AI's distinctive industry insight and forward-looking judgment about AI's move into scientific practice and even the real world.

UniPat AI also demonstrates a new model of AI collaboration:

Domain experts participate in task selection and review; partner laboratories provide unpublished tasks; and the project team organizes experiments from different disciplines into a unified set of evaluation tasks.

Take SPU as an example. Led by UniPat AI, its advisers are affiliated with institutions including Stanford, Cornell, Harvard, UIC, Peking University, the Hong Kong University of Science and Technology, and FutureHouse. Wet-lab collaborators from Youruisaisi, ABclonal, Deep Principle and Bota Bio participated in the relevant experimental work.

Cross-disciplinary knowledge and experimental resources thereby come together in a single evaluation process. SPU reflects the choice of research questions while also revealing what is needed to organize scientific questions into executable tasks: collaboration.

The 100 public tasks associated with SPU are currently planned for release as complete Harbor task packages. The 16 private tasks will remain private for controlled evaluation. The dataset and code are already open source.

Admittedly, today's AI is still only a novice when faced with this SPU mechanism.

It can use tools at lightning speed, install three software packages within eight hours, finish DFT and then run Monte Carlo. But when it comes to the most crucial judgment, it has not yet learned to ask itself whether the quantity it calculates is the one the experiment actually needs to measure.

But as tasks of this kind continue to accumulate, the kinds of scientific research work AI can undertake will also become clearer.

SPU has measured this distance. Every exam that follows will be a record of AI moving closer to the real laboratory bench.

Project homepage: https://spu.unipat.ai

Paper: https://arxiv.org/abs/2611.15049

GitHub open-source repository: https://github.com/UniPat-AI/SPU

This article comes from the QbitAI WeChat account. Author: QbitAI.

Related topics

After reading

Join the AINRK community

Read how to join the community and connect with people working in AI.

Community information (opens in a new tab)