AIE Summit: how small decision models support agent routing and verification

The author reviews decision-model cases from several companies, comparing latency, costs and errors while examining limits around finite options, threshold calibration and human review.
Editorial correction (November 13, 2026): Decision models can also make incorrect judgments and need application-specific calibration. Computational cost depends on context, generation length, caching and implementation; it cannot simply be described as necessarily growing quadratically.
On November 11, the second day of the AI Engineer Code Summit in San Francisco (AIE CODE 2026), the official theme was System One: small models that make judgments rather than generate content.

The author believes background agents need a judgment layer for narrow tasks that is low-latency and verifiable. "Right every time" is a goal, not a guarantee that current models have achieved.
Three points:
- Why it is needed: today's LLMs are all optimized as "assistants"—Einstein 95% of the time and Mr. Bean 5% of the time. If we cannot tell which is which on a given occasion, we cannot entrust them with automation.
- Who is building it: Typesafe released Jev on September 15; Cloudflare trained its own Clef over a weekend and released four models in three weeks; Google released Gemini Decision One on the day of the conference.
- Measured figures: ZoomInfo used it as a judge. For the same agent's falsely reported steps, the LLM judge caught one, while it caught 876. Glean's safety-check recall rose from 31.5% to 90.2%. UiPath customers replaced LLMs in workflows and made them about 50 times faster.
I. ZoomInfo: the judge hallucinates too
One of the most revealing talks was by Brian Kennedy of ZoomInfo.
He presented a real example. An agent reported, "The email has been scheduled for Tuesday." In reality, it failed to create a record in Salesforce, with the error message explicitly saying "do not falsely report success." It then created a to-do item instead. More importantly, this agent did not even have a tool for sending email.
The LLM judge responsible for scoring nevertheless approved it. Kennedy's explanation was that the judge received both the "story" and the "receipt," and believed the story.
ZoomInfo manually reviewed 500 real production steps. The original LLM judge had 65% accuracy—in his words, "roughly a coin toss." More than a quarter of the approved steps should have been rejected.
The change was to stop asking the judge whether a step was "done well," and instead ask a fixed set of yes/no questions at each step, answerable from the receipt. In one call, Jev answers a three-way choice—accept, retry or stop—plus seven yes/no questions. Steps with an acceptance probability of 30% or higher are approved, those at 5% or lower are rejected, and only the middle range goes to the LLM judge. Jev has sufficient confidence for about 70% of steps, with the LLM handling the remaining 30% as a fallback.
For that fake email, the LLM judge spent 2.5 seconds approving it; Jev took 0.2 seconds to return "accept 0%, retry 97%."

They ran Jev in parallel in shadow mode for 37,000 decisions. Its judgments did not affect actual operation and were used only for comparison:


The next step is to add "postcondition" checks: after invoking a sending tool, ask "Was it delivered?" After writing a record, ask "Does the record exist?" These would join the seven questions in the same call.
His implementation advice had three steps: choose a judgment and rewrite it as yes/no questions, run it in shadow mode alongside the existing judge, then examine where they disagree.
His closing line: "We trusted our judge until we measured it."
II. What System One is, and why LLMs struggle with judgments
Autoregressive models generate tokens one at a time, and generation length and context affect cost. The specific complexity also depends on the form of attention, KV caching and implementation. We cannot broadly claim that the total cost of every call grows quadratically with length.
Its place. UiPath positions it between deterministic code and an agent. Most of a workflow consists of deterministic calls, with a few clearly defined "cognitive decision points" in between. At those points, a model controls branching, rather than handing the entire workflow to an agent. Scott Roberts of Workday put it more plainly: from a coding perspective, it is an if-condition. It needs to be fast, smart, easy to use, general-purpose and very cheap.
Why LLMs fall short. Diogo Almeida of Typesafe—the company behind Jev, and someone who worked on ChatGPT, InstructGPT and RLHF—described a contrast: AI can solve Millennium Prize-level mathematical problems, yet cannot do accounting or handle fast-food drive-through orders well. The former are human-in-the-loop tasks, where satisfying a person is enough; the latter are automation tasks, where correct decisions are required. RLHF puts humans in the loop, so all current LLMs are optimized as "assistants" and cannot simply be run in the background.
He positions Jev around neither speed, cost nor its interface, but reliability. An LLM is "Einstein 95% of the time and Mr. Bean 5% of the time." If you cannot distinguish which it is on a given occasion, you cannot automate. He said his team had urged him not to say this publicly to preserve their long-term advantage, but he said it anyway.

He calls this "an even more bitter lesson": compute beats algorithms, data beats compute, and the bitterest part is choosing the right task. Jev is not a bigger model; it changes the task.
He offered another analogy: AI is software's fourth type of "semantic logic gate," allowing programming with common sense and intent. Instead of writing a regular expression, simply say, "I want a real Gmail address." The goal is logic-gate reliability—though he acknowledged that this is a high bar and much work remains.
His conclusion therefore runs counter to the popular narrative: the SaaS apocalypse will not happen; software will become stronger. In his view, "the root of slop" is AI's unreliability, which keeps companies from using it for decisions whose consequences they would have to bear.
III. Cloudflare trained one over a weekend; Google released one that day
Cloudflare's Michelle Chen gave a candid account of its history. When building Workers AI in 2023, Cloudflare bet that "models would get smaller and network latency would matter more." Instead, models grew from 7 billion parameters to 3 trillion, and it was only in March 2026 that the company began hosting trillion-parameter models such as Kimi, GLM and DeepSeek.
Jev launched on September 15. Cloudflare had its first experiment by September 18, then deliberated for about a week before deciding one Friday evening to proceed. Training finished in 48 hours over the weekend, and Clef launched the following week. Clef Flash followed, along with an extraction model that turns email, tweets, Slack messages and images into structured data: four model releases in three weeks altogether.

Why train its own? Jev is closed-source, so Cloudflare could not run it on its own network. Why is it useful? Her explanation was that these judgments need neither minutes of LLM reasoning nor a human in the loop whose schedule interrupts things: agents can run 24 hours a day. Her conclusion: "Cloudflare was not wrong; it was just early."
She demonstrated a CI-troubleshooting agent connected to 40 tools. First, System One cuts the tools to 10, extracts the PR number and pulls only errors from hundreds of log lines, before passing them to a large model. In this live run, large-model calls were six versus one and tokens 40,000 versus 1,800, with the same answer. Another run before the conference recorded 9.2 seconds versus 17.2 seconds and a 72% cost reduction. She stressed that this was illustrative: the goal is to reduce the LLM's workload, not replace it completely. Typical uses include routing, fraud prevention, spam and phishing judgments, and extracting data from web pages and support tickets.

Google DeepMind's Philipp Schmid opened his talk by announcing that about an hour earlier, Google had released the Gemini Decisions API, with a model named Gemini Decision One. It is natively multimodal, supports up to 1,000 questions per call, takes 150–250 milliseconds end to end, and costs $0.10 per million tokens.

Sponsors have followed suit too. Search API company Parallel launched WebJev, promoting it as "the first grounded decision model." It reads without generating, at a per-token reading cost of about 1/240 that of frontier large models.
IV. Measurements from each company
Glean (enterprise search; Chinmay Goyal)
- The selection rule in one sentence: if the possible outputs can be listed in advance, use Jev; if free text must be generated, return to an LLM. Fine-tuning is worthwhile only when labels are stable and volume is high; if labels change often, change the prompt.
- Query classification in the test environment: without training, Jev's quality was nearly twice that of the open-source Llama base model and about 90% of fine-tuned Llama's.
- Expert routing: latency fell to about one-eighth. He also acknowledged the trade-off: considerable implementation and infrastructure changes were needed, but the latency reduction was worth it.
- Ranking follow-up suggestions had a 1.5–2-second deadline: Jev returned results on time in 99.2% of cases, versus about 90% for a general-purpose LLM. Programmatic validation violations, such as using the wrong language, were about 10% for the LLM and zero for Jev.
- The LLM judge in evaluations was also replaced by Jev: only one result disagreed across 20–30 cases, and the team ran experiments 5–6 times faster.
- Safety checks: the original architecture was single-tenant, with a separate T4 GPU classifier for each customer—thousands of customers meant thousands of deployments. After switching to Jev, first-stage recall rose from 31.5% to 90.2%, costs fell by more than half, and the work of 2–3 full-time maintenance engineers was also saved.

UiPath (process automation; Chibi Vikram and Kevin Bernstein)
- The talk was titled "Your Automation Is Overthinking." In the fraud-prevention demo, a suspicious $48,000 transaction triggered a workflow. System One judged each of the most recent 60 transactions within one second and flagged four. It only judged whether the evidence was sufficient; only if not did it launch a large-model agent to call banking interfaces.
- Customers replaced LLM calls in workflows with Jev: about 50 times as fast, at 1/7.1 of the cost.

- They let a coding agent run an existing evaluation set overnight for 14 hours. It devised a "tournament" reranking method: document chunks were grouped for pairwise comparisons, and winners advanced. Thousands upon thousands of Jev calls still finished within one second; the score rose from 0.866 to 0.889, with roughly 15-fold greater speed. Their point was that an individual call may not look so smart, but the right strategy and combination can make existing models smarter.
- Classifying a million customer-service emails: first have an LLM propose about 15 candidate categories, then let Jev make a pass to see which buckets are full or empty. The LLM adjusts the categories, and they are finalized after a few rounds. Subsequent incoming emails use only Jev; the LLM leaves the process entirely.
Workday (Scott Roberts; the demonstration used synthetic data, and he explicitly said these features were not yet live)
- His view is that calling a large model at every agent step is an excess of intelligence; LLMs often act like someone holding a hammer and looking for nails.
- As a user types, Jev judges intent and brings up action cards for tasks such as checking a payslip, booking a meeting or finding someone. Seven cards plus "just chatting" achieved 96% accuracy, at about $20 per million judgments.

- Resume screening used no embeddings, vector database or reranking model. A SQL function in DuckDB called Jev directly to calculate each resume's "probability of meeting the criteria": 117 resumes in 2.3 seconds.
V. Where the boundaries lie
This layer is not all-purpose, and speakers made several limits clear:
- It only suits outputs that can be enumerated. Glean's rule marks the boundary: to generate free text, return to an LLM.
- In most scenarios, it reduces the large model's workload. Cloudflare's pipeline is "small-model extraction → small-model decision → large-model action." UiPath calls the large model only when evidence is insufficient. The LLM can leave entirely only for tasks such as classification with fixed categories.
- It also has times when it is uncertain. At ZoomInfo, Jev has sufficient confidence for only about 70% of steps, leaving the rest to an LLM. Latent Space has reported calibration problems with Jev: ask about the outcome of a fair die, and it assigns its answer about 83% confidence. The probabilities can be used, but thresholds must be set for each scenario. Glean, for example, learns thresholds separately for each customer.
- The ecosystem is still concentrated. Glean, UiPath, ZoomInfo and Workday all use Typesafe's Jev. Cloudflare trained its own precisely because Jev is closed-source, while Google directly built a similar product.

VI. In the context of the whole conference: cheaper judgments make verification possible at every step
Viewed across these three days, System One addresses the conference's biggest question: how to verify what agents write and do.
- Lauren Tan of SpaceX Superintelligence (the GrokBot team, which shipped 11,000 PRs last month) discussed another approach: each correction to an agent becomes a deterministic check. She quoted a developer who spends 70% of daily tokens on verification, saying she does too. Verification is already the largest token expense; making judgments 40 times cheaper means checking more, at every step.
- Simon Willison spoke at the closing session about an incident in July this year: a model OpenAI was evaluating found a sandbox vulnerability, bypassed the proxy and escaped to Hugging Face to cheat on a test it was stuck on. Initially, even OpenAI itself did not notice. Diogo's view on such incidents that afternoon was: "Detecting an agent doing something bad is much easier than doing it." Real-time interception is not possible today because judgments are not yet fast, cheap or reliable enough.
Taken together: write whatever can become a deterministic check as code; give System One what cannot be written as code but has enumerable answers; leave only the truly open-ended remainder to large models and people.
On the third day, main sponsor Google DeepMind had a talk by Ishaan Dey whose title perfectly footnoted this day: "Judgement is the Last Job."
Finally
The author's conclusion is that reliable, verifiable judgments are increasingly important. Existing models still make mistakes, and applications need calibration, checks and appropriate review.
In your agent workflow, how many large-model calls are really just answering a multiple-choice question?
If you have friends building agent products or enterprise automation, or following inference and the model sector, pass this along to them.
This article comes from "Lang Hanwei Will." Author: "Lang Hanwei Will."
Related topics
After reading
Join the AINRK community
Read how to join the community and connect with people working in AI.




