NEO-VLA studies how native visual representations support robot action learning

NEO-VLA lets an action expert read multiple backbone layers, comparing frozen models, fine-tuning and joint understanding training. Simulation and three physical tasks evaluate transfer, robustness and remaining spatial/task-generalization limits.
Placing a tennis ball securely on a can requires recognizing objects, judging alignment with the can’s top and knowing when to release the gripper. Such apparently simple actions depend on subtle visual information. But can action models use it fully?
Researchers from Nanyang Technological University’s S-Lab, SenseTime Research and Ropedia recently proposed NEO-VLA, bringing native vision-language models into robot action learning. After images are converted by lightweight convolutional layers, they enter the backbone together with language instructions. The module responsible for generating actions directly reads representations from different depths of the backbone. The entire visual pathway requires no separate pretrained visual encoder.
The team examined four questions: is a native backbone better suited to action learning, which layers matter most, how does retaining multimodal understanding affect training, and how does performance change across tasks and scenes? Controlled experiments and simulated/physical tests address them. The paper is public on arXiv.

- Paper title: From Pixels to Actions - Towards Native Vision-Language-Action Policies
- Paper: https://huggingface.co/papers/2611.06918
- WebPage: https://www.lmms-lab.com/posts/neo_vla/
- Github: https://github.com/EvolvingLMMs-Lab/NEO/tree/main/VLAKit
- Model: https://huggingface.co/weepiess2383/NEO-VLA
- DataSet: https://huggingface.co/datasets/weepiess2383/NEO-VLA-Action-Pretrain
How visual details reach action models
At present, most vision-language-action (VLA) models follow a similar processing pipeline: pretrained visual encoders such as CLIP, SigLIP and DINOv2 first extract image features, which are then passed to the language backbone and action module. Action decisions typically begin with the encoder’s output.
However, operations such as grasping, alignment and insertion need to use object edges, local geometry, depth, and subtle changes in the state of the gripper relative to the object. As visual information is progressively abstracted layer by layer, some cues relevant to fine manipulation may become harder to use.
Previous research has found that visual modules have a significant impact on VLA performance and are relatively sensitive to model pruning. Action fine-tuning may also damage pretrained visual representations. These observations suggest that reconsidering the visual–action interface itself.
NEO-VLA asks: if the action module can directly access representations at different levels as they form during visual computation, could it make better use of the manipulation-related information they contain?
Reading vision-language representations at different depths
By visual processing and action output, related architectures fall into three categories, illustrated below.

Figure 1 | Visual interfaces and action outputs in three VLA designs
Even when encoder-based VLAs read different language layers, their visual information has already passed through a separate encoder. NEO-VLA models vision and language together from layer one, exposing the process by which visual representations form. Which layer to read becomes a design question in its own right.
From observations to actions
NEO-VLA uses NEO-2B, a forty-layer native multimodal backbone with about two billion parameters and no separate vision Transformer, autoencoder or discrete visual codebook. A narrower action expert is added and trained from random initialization.

Figure 2 | Multimodal inputs, backbone and action expert
Unifying multi-view images, instructions and robot state
The model receives observations from both the previous and current time steps, with up to three camera views at each time. Each view’s input consists of a camera identifier, camera intrinsics and rotation information, and visual tokens obtained by dividing the image into patches. All views and time steps share the same image embedding layer. Images retain their original aspect ratios, and each view uses at most 88 visual tokens.
Following the visual input is a structured prompt specifying the control frequency, robot type, gripper configuration, action reference frame and task instruction. The model also uses six state tokens to represent the states of both arms and the body at the previous and current time steps. Corresponding inputs are masked for components a robot does not have, enabling the same interface to support different configurations, including single-arm and dual-arm robots and robots with mobile bases.
Layer mapping determines what the action expert reads
Each expert layer reads cached KV from a designated backbone layer. Four mappings are compared as follows:
- first-28: read layers 0–27 and truncate the backbone to those 28 layers.
- skip-28: retain all forty layers and select 28 approximately evenly.
- end-28: retain forty layers and read layers 12–39.
- full-40: read all forty with a forty-layer action expert.
The first three share the same 28-layer expert. skip-28 and end-28 also preserve backbone depth, isolating read positions most directly. first-28 changes compute by shortening the backbone; full-40 is separately considered as a larger-capacity expert.
Flow matching generates thirty continuous action steps at once
The action expert uses flow matching to predict an action chunk containing 30 time steps at once, using a unified representation for the movements of both arms and the body. During inference, the model generates actions by solving with 10 Euler steps by default. The backbone needs to be computed only once, and every subsequent sampling step reuses that same KV cache.
Three-stage action training
Training first adapts the expert, then updates the full model and introduces internally collected robot data.
Stage one freezes the pretrained backbone and trains the expert and conditioning modules for 113,400 steps with global batch 8,192.
Stage two unfreezes everything for 75,000 steps on the same robot data, adding a multimodal-understanding batch every other optimization step to preserve understanding.
Stage three starts from stage two and performs another 75,000 full-parameter steps on internal data, retaining other stage-two settings.
A unified data format across robots
2.23 million trajectories from nine public dataset categories
Pretraining combines AgiBot-World, Bimanual YAM, RoboCOIN, Open X-Embodiment, Galaxea Open-World, RoboMIND, RH20T, DROID and simulated InternData-A1. The complete corpus contains 2,231,583 trajectories, approximately 928 million data records, 2,704 scenes and 7,444 camera views, with recorded duration approaching ten thousand hours across single-arm, dual-arm and other configurations.

Table 1 | Corpus composition and sampling shares
Mapping different embodiments into one action format
Different robots do not record actions in the same way. To train them jointly, NEO-VLA converts their recordings to a unified action format and uses validity masks to distinguish the channels each robot actually has.
Each arm has 26 dimensions: seven joint targets, three translational increments, three rotation-vector increments, one gripper state and twelve hand-joint targets.
The body has eight: three for base motion, three for torso and two for head.
End-effector translation and rotation increments are expressed in centimeters and radians, respectively, in either the camera coordinate frame or the robot-base coordinate frame. Joint and hand targets use absolute values, while body actions use the base coordinate frame.
Actions are normalized separately for each robot embodiment. Components that do not exist, missing time steps and invalid channels are excluded from the training loss by masks.
Handling anomalies by level while retaining useful data
Public datasets differ in collection and annotation. The team defines five quality-check categories, handling readings, channels, action targets, trajectories and calibration information separately according to the level at which an anomaly occurs. This avoids discarding an entire trajectory that still contains useful information because of a localized problem.

Figure 3 | Five quality checks and their processing scopes. Three experimental findings follow.
A frozen native backbone already supports effective action learning
To compare representations, identical expert designs were attached to NEO-2B, Qwen3-VL-2B and Gemma-3-4B, pretrained for 75,000 steps on Open X-Embodiment under identical settings, then evaluated on LIBERO.

Table 2 | Controlled LIBERO backbone comparison
With the backbone frozen and only the expert trained, NEO-VLA averaged 90.9%, above Qwen3-VL’s 79.5% and Gemma-3’s 84.5%. It approached their full-fine-tuning scores of 90.8% and 91.0%; on LIBERO-Long, frozen NEO-VLA reached 85.4%, exceeding both fully fine-tuned comparators.
This indicates NEO’s pretrained representations transfer well to action.

Table 3 | Qwen3-VL expert access to intermediate visual features
Does this advantage come solely from access to intermediate visual features? The researchers conducted an additional controlled comparison with Qwen3-VL, directly connecting 13 layers in the action expert to intermediate features from the visual encoder without adding parameters. With the backbone frozen, this connection reduced the success rate from 79.5% to 64.9%; after full fine-tuning, new and original interfaces were nearly identical at 90.9% and 90.8%.
Thus, intermediate-feature access alone does not reproduce the benefit. These results suggest that representations formed through native vision-language pretraining are more conducive to action learning. How representations are learned and how they are read need to be considered together.
The different backbones still differ in their pretraining data and model capacity, so the entire improvement cannot be attributed to the architectural change of removing the visual encoder. In this set of comparisons, however, the native backbone showed relatively strong action-transfer capabilities both when frozen and when fully fine-tuned.
Action prediction is especially sensitive to early layers
Comparing layer selection and freezing first showed that early connections are important.
Researchers then intervened at each read position to identify influential information.
First, the KV representation at a particular read location was replaced with the representation from another observation. Replacements at layers 0 through 10 caused the largest increases in action translation prediction error. As a control, the researchers introduced noise perturbations with matched amplitudes; the resulting error differed from the unmodified baseline by less than 0.004 centimeters. This suggests that the model’s sensitivity to these layers is primarily related to the content of their representations.


Figure 4 | Responses to representation replacement and input masking at different depths
Action information is distributed across depths. Early representations are particularly important when frozen; after full fine-tuning, combining information from multiple depths works better. The effects of language, appearance and geometry information on action prediction also show different distributions across depths.
Retaining understanding data matters during action post-training
What happens to the model’s original multimodal understanding capabilities after it learns actions? The team compared two post-training approaches: using action data alone, and jointly training on action data and general multimodal understanding corpora. The model before the start of post-training served as the reference.

Table 4 compares joint understanding-and-action training with training on action data alone. The model trained only on action data cannot generate valid text, so its results on five understanding benchmarks are marked with “—”.
Joint training broadly preserves MMBench-EN, RealWorldQA, AI2D, TextVQA and ChartQA. Four decline no more than 1.5 percentage points, while RealWorldQA rises 1.6 points. RoboTwin 2.0 Clean2Rand success reaches 69.4%.
By contrast, the model post-trained only on action data can no longer generate valid text, and the paper marks its results on all five understanding benchmarks with “—”. Its Clean2Rand action success rate also falls to 10.7%.
The results indicate a close link between multimodal understanding and robust action. Continuing to include understanding data during action post-training can both preserve the backbone’s existing understanding capabilities and help the policy adapt to randomized scenarios.
From standard benchmarks to scene changes and physical operation
The earlier controlled experiments were used to analyze the model’s design; the following section examines the complete model’s performance on different tasks. Simulator versions, camera configurations, training data and evaluation protocols are not fully consistent across the different studies. Scores compared across papers therefore mainly provide a reference and should not be treated as strict comparisons under identical conditions.
LIBERO
The complete model averages 97.4% success on LIBERO. At present, the scores of several methods on this benchmark are already relatively close. This result therefore mainly shows that NEO-VLA maintains strong in-distribution manipulation capabilities. Its training and evaluation settings differ from those of the backbone comparison experiments discussed earlier.

Table 5 | Complete-model LIBERO results
LIBERO-Plus tests seven perturbation categories

Table 6 | LIBERO-Plus perturbation results
The results show that NEO-VLA adapts better to changes in observation methods and image conditions, echoing the earlier analysis of shallow visual representations. Changes in embodiment or spatial configuration remain more difficult problems.
LIBERO-Pro exposes spatial and task-generalization difficulties
On LIBERO-Pro, NEO-VLA scores 98.0 under changes in instruction semantics and 84.3 under changes in object appearance. It performs poorly, however, when object positions or task goals change; other methods also generally struggle with these two types of test. This gap shows that stronger perceptual and semantic robustness does not directly translate into flexible adaptation to spatial arrangements and tasks.

Table 7 | LIBERO-Pro appearance, location, semantics and task generalization
RoboTwin 2.0 tests randomized environments
Under the official 2,500-demonstration setup, Clean scores 75.78 and Clean2Rand 69.40, a 6.38-percentage-point decline.

Table 8 | RoboTwin 2.0 clean and randomized scenes
Fifty demonstrations per physical-task fine-tuning
Three physical tasks test precise placement, multistep manipulation and object-pose adjustment: placing a tennis ball on a can; putting tape in a bowl then a ball inside the tape; and standing a horizontal can and inserting it into tape. Each uses fifty fine-tuning demonstrations and twenty tests, split equally between in- and out-of-distribution initial conditions.

Figure 5 | Three quantitative physical tasks
With identical physical data but each method’s recommended post-training, NEO-VLA succeeds 12, 17 and 13 times; StarVLA 1, 2 and 0; and π0.5 10, 14 and 13. NEO never scores below π0.5 on these tasks.

Table 9 | Success counts out of twenty trials per task
In addition to the three quantitative tasks above, the team demonstrates pouring water and picking four pieces of gunkan sushi, one at a time, from a mixed selection and arranging them on a plate. These supplementary demonstrations illustrate the types of operations the model can perform; they are not included in the success-rate statistics for the first three tasks.

Figure 6 | Physical demonstrations and key action stages
About 0.26 seconds per chunk on one RTX 3090
The team also measured inference latency on the physical Pika robot system. The policy runs on a single RTX 3090 with 24 GB of video memory using BF16 precision, and each request returns an action chunk containing 30 steps. The experiment executes the predicted actions directly, without using RTC.

Table 10 | Model-forward and physical-request latency in milliseconds
Turning perception and understanding into more flexible actions
The experiments show that action learning is influenced both by how visual representations are formed and by whether the action module can read information from different stages of visual computation. The native backbone can already support action learning effectively while frozen, and access to shallow representations is particularly important. At the same time, directly reading intermediate features from a conventional visual encoder did not bring the same gains. These results suggest that learning representations and exposing them to action modules must be considered together.
The model has demonstrated some ability to adapt to changes in appearance and observation conditions, but new spatial layouts and task goals still pose significant challenges. The next question is how to turn its existing perception and understanding capabilities into more flexible actions, enabling the robot to adjust its manipulation strategy even when geometric relationships or task intentions change.
This article was originally published by Synced. Author: Synced.
Related topics
After reading
Join the AINRK community
Read how to join the community and connect with people working in AI.




