AI guides

Deepnote benchmarks proactive AI coworkers on intervention judgment, scores and model-call costs

By 新智元1 viewsSource: 新智元
Deepnote compares intervention judgment, scores and model-call costs for proactive AI coworkers

Twelve models chose whether to respond across 50 simulated workplace tasks. Astra and Sol led the reported scores; monthly costs are estimates extrapolated from assumed traffic.

Editorial correction (November 16, 2026): The leaderboard scores come from a limited set of simulated tasks and model judges. Costs were recorded under the models, prompts and invocation settings used; monthly enterprise costs are extrapolated from traffic, not measured bills.

Pay 430 times as much for a score only 11% higher!

That is staggering. The data collaboration platform Deepnote today released the results of an evaluation—

Founder Jakub Jurovych created something called the Agent Initiative Benchmark, putting 12 frontier models into the same company's Slack channels as AI coworkers and letting them decide for themselves when to help and when to keep quiet.

And after the runs—OpenAI took both of the top two spots!

GPT-6 Astra scored 8.97 points, taking first place. GPT-6.1 Sol scored 8.65 points, coming second, at a cost of just $0.11 per task.

Anthropic's Fable 5.2 came third, with 8.17 points and a cost of $1.87 per task.

Seventeen times the price, and a lower score.

Article illustration

Converted into a monthly Slack-monitoring bill for a 70-person company: $281 for Sol and $4,684 for Fable 5.2.

For the same work, Fable 5.2 costs 16 times as much.

But the most outlandish comparison in the entire table is this: Fable 5.1 costs $3.02 per task, while GPT-6 Luna costs $0.007. The prices differ by a factor of 430, but the scores differ by only 1.1 points.

Think about that.

Article illustration

What exactly is this benchmark testing?

It does not test how smart a model is. It tests whether the model is reliable as a coworker.

Deepnote created a fictional company called PageMantle.

Yet this “fiction” is extraordinarily realistic: six years of operating history, 71 employees, 100 enterprise customers and a full set of 11 business systems, from CRM, invoices and tickets to product analytics and a data warehouse.

Article illustration

Even real-world events such as the collapse of SVB and competitors releasing new features were incorporated into the company's timeline. Employees would discuss response strategies in Slack, then open issues in Linear to follow up.

The AI's role is to lurk in Slack, listen in on every conversation, and decide for itself whether to step in.

There are three ways to intervene: reply in the channel, reply in a thread, or send a direct message to the person concerned. Of course, there is also a fourth—do nothing.

And that fourth option is precisely the hardest part.

Fifty realistic workplace tasks cover eight kinds of scenarios: finding information, explaining processes, analyzing data, checking claims, making judgments, diagnosing problems, coordinating teams and reviewing security.

Each of the 12 models ran through the tasks once, with a three-minute limit per run. Two blinded model judges independently assigned scores from 0 to 10. There were 600 complete runs, and the dataset and code are both fully open source.

Here is what the data show:

Sol and Astra scored 8.65 and 8.97, respectively, on this ten-point scale, with overlapping confidence intervals. Ratios between scores cannot be directly converted into ratios of general capability. Differences in per-task costs also apply only to this configuration.

Second, Anthropic is expensive across the board. Anthropic holds five of the top 12 places—Fable 5.2, Opus 5.5, Fable 5.1, Sonnet 5.5 and Haiku 5.5—but none makes the top three when ranked by value for money. Third place for Fable 5.2 looks good, but its per-task cost is 17 times Sol's, and its monthly bill is 16.7 times Sol's. The only consolation is speed: Sonnet 5.5 takes just 15 seconds and Opus 5.5 just 29 seconds, making them the two fastest models in the field.

Article illustration

Third, Gemini and Grok fell flat. Both averaged 35 tool calls per task, twice as many as Sol. But more calls do not mean better work: Gemini timed out without submitting an answer on 27 of the 50 tasks, and Grok on 18. When exam time is up and the answer sheet is still blank, the scores naturally end up at the bottom.

Article illustration

The hardest part is not helping, but knowing when to keep quiet

The benchmark's toughest design choice was to include ten scenarios in which the AI “should not help”: the matter had already been resolved, or it was simply none of the AI's business.

The results are interesting: when it comes to helping, all the models were eager. They could handle 90% of the 40 tasks that called for intervention.

But when they should keep quiet? Even the best performer held back only eight times, while most held back only six times.

GPT-6 Luna, DeepSeek V4.1 Flash and Mistral Large 4 spoke up in six of the ten scenarios that called for silence. Both Fable models also spoke up four times.

The point is that these “things that should not have been said” were actually quite high-quality. The models diligently searched systems, checked data and wrote analyses, then solemnly sent a reply to a message that needed no reply.

It is like that coworker who speaks at every meeting. The problem is not competence; they have not worked out whether they are actually needed in this matter.

And being “overly helpful” costs money. Deepnote found that when the same model intervened inappropriately, it spent, on average, 1.8 times more time and money than when it stayed silent. The model is not simply speaking out of turn off the cuff; it speaks out of turn only after diligently working through a full round of tool calls.

This also explains the divide between two model styles, which Deepnote calls “snipers” and “deep thinkers”:

  • Representative “sniper”: Opus 5.5. An average of ten tool calls, an answer submitted in 29 seconds, and a score of 7.84. Fast, precise and decisive, with no unnecessary talk. It also held back in eight of the ten scenarios that called for silence, tying with Astra and Sol for the best result.

  • Representative “deep thinker”: Sol 6.1. An average of 18 tool calls, an answer submitted in 54 seconds, and a score of 8.65. It took 80% more time and made 80% more tool calls, but scored 10% higher and cost just one-tenth as much as Opus.

Article illustration

Article illustration

Gemini and Grok belong to a third type: the lost ones.

Thirty-five tool calls, yet no answer in time on half the tasks. They kept leafing through the books until the exam ended, with the answer sheet still blank.

Why does this matter?

First, the criteria for choosing AI coworkers have changed.

Traditional benchmarks compare “who is smartest.” But when AI's role shifts from “you ask, I answer” to “a proactive coworker online around the clock,” the criteria become:

be able to help when help is needed, stay quiet when it is not, and spend an amount that stands up to scrutiny. This is judgment, not intelligence.

OpenAI's two models currently do best across this triangle. Sol stands out in particular: it almost reproduces Astra's judgment—helping on 39/40 tasks and staying silent on 8/10—but at just one-tenth the price.

Second, the billing logic for always-on agents is completely different.

Previously, you paid once for each API call and left when you were done.

But an AI coworker online around the clock incurs continuous charges based on traffic. Sol costs $281 a month, Fable 5.2 costs $4,684 a month, and Fable 5.1 costs $7,540 a month.

These monthly costs are extrapolations based on assumed traffic for a team of around 70 people, not bills paid by a real company. Actual invocation frequency, caching, retries and task length will all change spending.

If your company has 500 people, and if the agent also has to monitor email, calendars and tickets at the same time, the savings are no longer pocket change: they amount to a real staffing budget.

Third, a new path has emerged for trading off speed and quality.

Anthropic is fastest (Sonnet, 15 seconds), OpenAI is most accurate (Astra, 8.97 points), and Sol offers the best value ($0.11).

For teams building products, the answer may not be “choose one,” but “route by scenario”: use Anthropic's fast models to save time in real-time conversations, and use Sol for background analysis to save money while achieving high scores.

Deepnote also recommends this tiered strategy in its report.

The most important ability for an AI coworker is not how smart it is, but how dependable it is: speak accurately when it should speak, resist the urge when it should stay quiet, and keep the bill from becoming too outrageous.

Under this set of tasks, judges and cost definitions, Sol shows a relatively good combination of scores and costs. This cannot establish an upper limit on value for money across all teams, model versions and workloads.

And those models that cost 17 or 430 times as much without earning many more points have something to think hard about: in the world of AI coworkers, the most expensive one is not necessarily the one the boss most wants to keep.

References:

https://x.com/jakubjurovych/status/2121256499552915586

https://deepnote.com/research/benchmark

This article comes from the Xinzhiyuan WeChat account. Author: Xinzhiyuan.

Related topics

After reading

Join the AINRK community

Read how to join the community and connect with people working in AI.

Community information (opens in a new tab)