AI guides

Neos Co-Trains Search Policies and Evidence Interfaces, Reporting Gains on Ten Multimodal Benchmarks

By 量子位0 viewsSource: 量子位
Neos jointly trains search policies and evidence interfaces, reporting gains on ten multimodal benchmarks

The Accio-led team combines visually dependent tasks, process credit assignment and alternating training. Neos-35B averages 64.7, still 2.4 percentage points behind the comparison Gemini workflow.

For researchers working on multimodal deep search, the most troublesome problem is usually continually obtaining new visual evidence while avoiding redundancy between text and image information.

An ideal search system should precisely locate multimodal evidence based on the user's description, and efficiently filter and integrate information from the web. By providing agents with concise, high-quality evidence, even models with relatively few parameters may be able to achieve search capabilities close to those of leading closed-source models.

With this in mind, a research team from Alibaba's Accio team, Fudan University and the Shanghai Innovation Research Institute proposed Neos, bringing data construction, process rewards and co-evolution between the agent and its environment into a unified training framework.

Article illustration

Article illustration

△ Performance of Neos-9B, Neos-35B and other methods on four representative search benchmarks.

The team designed the following targeted experiment: give an agent a photograph of a band performing and ask it to find the number of knobs on the tape recorder the band used when recording a particular album in 1967.

Neos first identified the Beatles through reverse image search, then followed information about the album's recording to the Studer J37. The first crop confirmed five knobs. Reviewing the previous search results, the agent noticed what appeared to be a separate knob on the right side of the machine. It continued consulting webpages and gathering additional photos, eventually confirming that there was one more on the right and arriving at the answer, '6.'

Article illustration

△ Starting from a concert photograph, progressively obtain and verify the visual details of the tape recorder's panel.

This approach of continuing to fill gaps in the evidence and improve accuracy during a search is precisely the focus of Neos's research.

Data construction: make intermediate visual evidence necessary to solve the task

Starting from a knowledge graph, Neos constructs multi-hop reasoning tasks and places a visual entry point and intermediate visual checkpoints within the reasoning structure. The model must first identify objects or details in an image before it can continue tracing subsequent relationships, ensuring that visual information obtained during the search genuinely affects the next decision.

Article illustration

△ Construct visual dependencies, verify images, filter out shortcuts, and then collect verified expert trajectories.

Image verification uses a blind setting: the model inspecting the image cannot see the target's identity or the expected answer. It can only describe content based on the full image and local crops, then match it against a set of candidates containing distractors. Question rewriting subsequently removes names that reveal entity identities and hints about operations, while retaining the necessary relationship constraints and visual dependencies.

A four-stage filtering process further removes shortcuts. The system successively tries text-only direct answering, text-only search, VQA using only the entry image, and multimodal search with a short budget. If any one of these methods can solve the task, the candidate task is discarded. The first three checks screen out paths that bypass the acquisition of intermediate visual evidence; the final check controls task difficulty.

Questions that pass the filters are given to an expert model to solve in a real search environment. Verified successful trajectories are used for policy SFT and as references for subsequent credit assignment, while their local tool interactions are used to train the evidence interface. In this way, the same investigation trajectory can provide supervision for both decision-making and evidence processing.

Credit assignment: successful trajectories can explore freely, and failed trajectories can teach something too

Rewarding only the final answer can easily conflate correct searches, useful observations and critical errors within a failed attempt. Neos's reference-guided credit assignment handles successful and failed trajectories separately: successful trajectories retain the usual group-relative advantage to encourage diverse solution paths, while failed trajectories use verified successful references to identify the steps where the investigation genuinely went off course.

Article illustration

△ Identify critical errors, decay their downstream influence, and retain positive rewards for useful exploration and error correction.

One way to understand this is: 'Let good students have free rein; those who answered incorrectly first use a successful solution to correct their thinking.' The judging model considers the evidence that was visible at each step to distinguish cases such as identifying the wrong object, misreading an image, or continuing a search based on an existing error.

Critical errors are penalized, and their downstream influence decays with distance; useful exploration and self-correction are rewarded. Once the model corrects an error, propagation of the earlier error stops. This allows valuable local actions in a failed investigation to be reinforced, such as finding a key image, posing an effective query, or proactively gathering additional evidence.

MEI and co-evolution: let search policies and the evidence interface learn together

Webpage and image retrieval often return large amounts of redundant content, which auxiliary models need to filter and compress before passing it to the policy. Neos designs this layer as a trainable Multimodal Evidence Interface (MEI), embedded in WebVisit, ImageSearch and ReverseImageSearch, so evidence processing can continually improve alongside the search policy.

Article illustration

△ Policy RL and MEI SFT are updated in alternation, with online interactions providing both kinds of training signal.

When calling a tool, the policy states the local question it currently wants to resolve. Around this objective, MEI extracts webpage evidence, filters and ranks images, or judges visual matches, then returns structured text and image references. The policy uses this to choose its next action, continuing to retrieve images, crop them and verify them when needed.

Neos uses Qwen3.5-9B and Qwen3.5-35B-A3B as its policy backbones, while MEI uses a separate Qwen3.5-9B. After supervised initialization, the two enter alternating training: the policy undergoes RL based on answer correctness and process credit, while MEI selects effective calls from online interactions for SFT. Successful trajectories contribute effective MEI calls; failed trajectories contribute only calls that help advance the investigation or correct errors.

Changes to the policy create new evidence requirements, while MEI updates change the observations available to subsequent searches. The two adapt to one another through online feedback, forming a loop of obtaining evidence, adjusting decisions and improving evidence processing.

Ten benchmarks: small models approach frontier performance

Neos's evaluation covers factual visual question answering, multimodal retrieval and complex browsing tasks, comparing direct answers without tools, integration into a search workflow, and performance after specialized training.

Article illustration

The average scores of Neos-9B/35B reach 60.7/64.7, improving on their respective workflow baselines without specialized training by 17.4/15.5 percentage points. Of these, Neos-9B has already exceeded DeepVoyager-VL-30B-A3B's score of 58.6.

On complex search tasks, Neos-35B's scores on MMSearch-Plus, MM-BrowseComp and BrowseComp-V3 are respectively 52.7, 48.3 and 47.0, improvements over DeepVoyager-VL-30B-A3B of 12.1, 17.8 and 12.0 percentage points. Its ten-benchmark average trails the Gemini-3.1-Pro workflow's 67.1 by only 2.4 percentage points; on BrowseComp-VL, its score of 66.4 matches that of the GPT-5.6-Sol workflow.

Three key ablation studies

Data construction: With only 1K synthetic examples, the model exceeds a model trained on 1.1K REDSearcher-MM examples on BrowseComp-VL, MMSearch-Plus and FVQA by 1.5, 4.3 and 1.0 percentage points, respectively.

Credit assignment: GRPO using only final-outcome rewards scores below cold-start SFT on all three benchmarks. Compared with GRPO, the full process rewards improve results on BrowseComp-VL, MM-BrowseComp and FVQA by 3.6, 4.2 and 5.0 percentage points, respectively.

Co-evolution: Compared with freezing an already adapted MEI, alternating training still improves results on MMSearch-Plus, BrowseComp-V3 and VDR-Bench by 0.7, 0.7 and 1.0 percentage points, respectively, demonstrating the additional benefit of continually adapting to the policy.

Article illustration

About 32% fewer tool calls

From the first to the last RL update window, the average number of tool calls per Neos-9B trajectory fell from 29.7 to 20.2. Meanwhile, WebVisit's share of calls rose from 21.2% to 32.1%, while the combined share of image search, reverse image search, image retrieval and cropping rose from about 19.1% to 23.4%. Overall calls decreased, while the relative weight of reading source pages and verifying visual evidence increased.

Article illustration

△ During training, tool calls become more focused, and operations related to visual evidence account for a larger relative share.

From identifying the band to verifying the tape recorder's final knob, Neos demonstrates the potential of small models in sustained visual investigation: bring key images into the reasoning process, give useful exploration a training signal, and then let the search policy and evidence interface improve together.

GitHub: https://github.com/Accio-Lab/Neos

Homepage: https://Accio-Lab.github.io/Neos/

This article comes from the WeChat public account QbitAI and was written by QbitAI.

Related topics

After reading

Join the AINRK community

Read how to join the community and connect with people working in AI.

Community information (opens in a new tab)