AI news

Dahua Lin on Language and Multimodality: Why Spatial Perception Still Needs Long-Term Research

By GaKi0 viewsSource: 十字路口Crossing
Dahua Lin discusses language, multimodality, and spatial intelligence.

Dahua Lin discusses language's information density, visual detail, and spatial perception, explaining SenseTime's multimodal research strategy. The report also notes unresolved training challenges and uncertainty over how other modalities can advance intelligence.

Everyone is searching through the fog for the direction of next-generation AI.

Article illustration

Several leading AI companies seem to have reached different judgments about where the next generation of AI should go.

Anthropic has previously explained publicly why Claude has not invested in image and video generation as its competitors have. The reason is that the company wants to devote more resources to AI compute allocation, safety research, and AI agent research, focusing on improving Claude's capabilities in coding and complex work tasks.

Article illustration

Main visual on Anthropic's official website

On November 13, OpenAI made another adjustment to its robotics team, bringing the relevant researchers into its core research division. Although OpenAI subsequently clarified that robotics research and development was continuing, the adjustment also reflects its exploration of having general-purpose models take on more robotics tasks directly.

Meanwhile, Google DeepMind has continued to advance multimodality and understanding of the physical world.

Everyone wants to make AI more intelligent, but their judgments about where to invest resources next are not entirely the same; one could even say they differ sharply.

Chinese AI companies are also exploring different technical approaches. They are all considering how to combine capabilities such as language, vision, and video, and whether multimodality can become the key to improving AI intelligence in its next stage.

Recently, SenseTime co-founder and chief scientist Dahua Lin also discussed his views on language, multimodality, and the next stage in the development of intelligence in detail during a public talk.

Article illustration

🚥 As everyone searches through the fog for the direction of next-generation AI, SenseTime's Dahua Lin has offered his own judgment from the "crossroads of language and multimodality."

We have compiled the six most noteworthy judgments from Dahua Lin's remarks to share with you.

These six views explain why SenseTime has chosen this path while also addressing some of the core debates in today's AI industry.

This Interview Addresses the Debate Between Language and Multimodal Approaches

[1] What Dahua Lin Says About the Debate Between Language and Multimodal Approaches

The view that "multimodality can easily lower AI's IQ" is widely circulated in the LLM industry, and Dahua Lin did not reject it outright.

This is the starting point for the entire debate over language and multimodality.

Article illustration

Lin believes language models have progressed faster mainly because years of engineering experience have already accumulated. Relatively mature methods are available for reference on how to mix data, train models, and evaluate results.

By comparison, native multimodality still has many problems to solve.

Training a single model to both understand and generate different modalities, including text, images, video, and audio, is harder and more expensive.

But Lin believes multimodality has a very high ceiling and still has considerable room to develop.

Taking GPT-6's impact on the embodied intelligence industry as an example, he argues that embodied intelligence is just one application of multimodality, with many more possibilities ahead. This is also why he is optimistic about multimodality's long-term development.

Article illustration

GPT-6 Astra controls a robot to draw the Golden Gate Bridge by hand

But there is still a key question at present:

How can different modalities be incorporated into the same model without affecting its existing level of intelligence?

Lin believes multimodality itself does not make models less intelligent. Rather, sufficiently good training methods have yet to be found, and more time and investment are needed.

This also explains why current multimodal models still fall short of language-only models on some tasks.

[2] Language Has a High Density of Intelligence, Yet Even 2,000 Characters Cannot Recreate the Same Face

Lin's position on "why we cannot work on language alone" falls between the two camps. He does not endorse purely visual intelligence, nor does he believe language is already sufficient. SenseTime's starting point for multimodality is therefore to make up for information that language cannot express. Replacing language with vision is not something he is considering.

This is the central rationale for SenseTime's multimodal approach.

Article illustration

Lin also mentioned another view in academia:

Some young scholars believe that before humans had language, they could already observe their environment, hunt, and survive through vision. This suggests that intelligence does not necessarily have to depend on language, so they advocate developing purely visual intelligence.

Lin takes a different view.

He believes language is the modality with the highest information density.

As humans developed language, they gradually organized vast amounts of knowledge and experience into information that could be expressed, transmitted, and accumulated. The rapid development of today's language models is also connected to this accumulated knowledge.

But language has limitations of its own.

Lin gave a particularly interesting example:

Take a photograph of a person, ask the best multimodal model to write a 2,000-character description of their appearance, and then use an image generation model to draw them. What you get is a different person.

The reason is simple: a person's features, expression, and facial characteristics contain a great many details that are difficult to describe accurately in words. Even with 2,000 characters, it is difficult for another model to recreate the same face.

The same is true in daily life. If you want to know what someone looks like, a glance at a photograph is often more direct than reading thousands of characters describing their appearance.

This shows that although language can convey knowledge efficiently, much visual information is difficult to convert fully into words.

On this basis, SenseTime proposed the concept of a "spectrum" in an attempt to explain how different modalities differ in the way they express information.

Article illustration

The arXiv paper "The Prism Hypothesis": images and text are decomposed onto a unified spectrum. (Dahua Lin is one of the authors; the work is a collaboration between SenseTime Research and Nanyang Technological University.)

Language is better at expressing knowledge that has been organized and compressed, while images retain a wealth of detail that is difficult to describe in language. Each has its own strengths.

For another example, language is usually more efficient for programming and logical reasoning, whereas images are more direct for understanding spatial structures, recognizing people, and assessing visual details.

Lin believes truly complete intelligence requires drawing on the strengths of both language and vision, enabling models to understand abstract knowledge while also perceiving information that is difficult to express in language.

[3] In a Photograph With Six Fingers, the Model Counted Five

Most earlier multimodal models first translated images into semantics and then passed them to language for reasoning. Answering that six fingers were five shows that the models were using common knowledge in language to "guess" at the image rather than seeing the image itself clearly.

Article illustration

Lin's team first tested tasks that models such as GPT and Gemini performed poorly on, then went further to determine whether adding data could solve those problems. If adding data could solve them, the results would primarily demonstrate application value and would be unlikely to support innovation at the level of underlying principles.

In tests of perspective-taking (viewing the same object from different viewpoints), they also discovered an "anti-scaling law" phenomenon: after more relevant data was added, models actually became more prone to errors because they increasingly relied on semantics to understand what they saw.

For example, when people look at the same object from different angles, they naturally understand the changes in its appearance; the farther away they are, the smaller the object appears.

But the multimodal models at the time, including the best-performing Gemini, lacked an understanding of both scale and viewpoint. What they had primarily memorized was the correspondence between images and text.

Another example makes this a little more tangible:

When GPT, Gemini, and Grok were shown a photograph containing six fingers, they all answered five. This was because the models first recognized a "hand" and then answered based on their learned knowledge that "a hand has five fingers," rather than accurately identifying the actual number in the image.

Lin also noted that the latest models may already have solved this problem. But the phenomenon exposed a deeper issue: models rely excessively on converting visual information into semantics and then reasoning from those semantics.

Therefore, adding visual input to an LLM does not mean that the model truly understands the visual world.

Article illustration

The SenseNova-SI paper: a previous open-source SOTA model still scored 50.7 without being given an image, almost the same as with an image. This indicates that it relied on language priors rather than genuinely performing visual reasoning.

This is also why SenseTime has made spatial intelligence an important direction for exploratory research.

[4] Better Video Quality Does Not Mean a Model Is More Intelligent

Multimodality is often equated with "being able to generate images and video." But Lin believes the value of multimodality for intelligence lies in taking part in thinking, while generation affects only visual quality.

Article illustration

One view in the industry holds that multimodal generation itself cannot improve a model's level of intelligence.

Lin, however, judges that simply improving a model's generation capabilities mainly improves visual quality (Visual Quality). But incorporating spatial perception into a model's thinking process could change the way it reasons.

This may not make a model better at solving math problems, but it can give it more complete intelligence.

SenseTime's exploration of multimodality therefore focuses on unifying understanding and generation, allowing spatial perception to take part in the model's reasoning process as well.

Article illustration

Diagram of SenseTime's NEO-unify architecture: a native vision-language model directly processes pixels and text

This is also why Lin is optimistic about multimodality.

He believes AI's development across different fields over the past three years has been uneven. Language models have already surpassed top humans in tasks such as mathematics and programming, yet robots cannot even independently set up the filming equipment at the interview venue.

AI still has many capabilities to fill in as it moves from the digital world into real physical space.

Of course, there is still no clear answer as to exactly what modalities beyond language can bring to intelligence, or what roles they and language should each play.

Lin also candidly acknowledged that this is precisely the question his team is most concerned with and continues to study.

[5] A New Idea Appears in the United States, and Ten Chinese Companies "Race to Implement It" Within Two or Three Months

The AI industry's mainstream path is defined by a few American companies, with its focus on LLMs.

Following the mainstream and making optimizations produces results quickly. But when everyone is doing the same thing, long-term value is lost. Lin used this point to explain why SenseTime has not put all its resources into the mainstream approach and has instead chosen the less mainstream path of multimodality.

Article illustration

Lin believes open source has accelerated the spread of technology. Once a technical approach is known to be feasible, Chinese teams can usually find a concrete way to implement it very quickly.

For example, teams are rapidly following up on both RLVR (reinforcement learning based on verifiable outcomes), used in DeepSeek-R1, and the Recurrent Transformer that outside observers speculate GPT-6 uses.

Article illustration

DeepSeek-R1-Zero's AIME accuracy continues to rise during reinforcement learning training

This is because the underlying principles of many technologies have much in common, and people in the industry already have guesses about the broad approach. Once the direction is confirmed, what remains is to fill in the details through experiments, avoiding the process of trying out a large number of approaches one by one.

Lin calls this practice "racing to implement".

However, he also has another perspective on it.

When a new technology is released in the United States, a group of products using similar approaches may appear in China within two or three months, as everyone tries to claim a place in the ecosystem as quickly as possible.

If just one company did this, it might establish an advantage. In practice, however, ten companies are often doing similar things at the same time.

When everyone is doing the same thing, it is hard to build a long-term competitive advantage.

In Lin's view, a company's long-term competitiveness comes from "sustained compounding". The real difficulty is whether you can keep going when your chosen direction is less popular and lies outside the mainstream.

Only by continually investing in one direction and accumulating sufficiently deep technology and experience can you make it difficult for latecomers to catch up. Of course, he also agrees that optimizing and making incremental innovations along the mainstream path has considerable industrial and technological value.

But if China wants to make a longer-term contribution to AI, it needs to exercise its own technical judgment and produce things that are truly different.

This is the logic behind the judgment to "remain committed to multimodality."

[6] One-Fifth of Resources Are Not Tied to Any Product

Staying committed to multimodality requires resources and organizational support. SenseTime has chosen to continue putting 20% of its resources into exploratory research, and spatial intelligence emerged from that investment.

Article illustration

Lin specifically noted that SenseTime still devotes 20% of its resources to exploratory and forward-looking research that is not tied directly to a product.

However great its business pressures, SenseTime reserves some of its resources for directions that, based on first principles, it considers worth studying over the long term, such as spatial intelligence and the unification of understanding and generation.

These studies do not generate revenue directly in the short term and have no definite product plans. The spatial intelligence "anti-scaling law" phenomenon mentioned earlier was discovered through this kind of research.

Lin also acknowledged that there is an order-of-magnitude gap between SenseTime and OpenAI in resources.

Article illustration

Main visual on the OpenAI GPT-6 Astra release page

But in his view, GPT-6's progress demonstrates the value of this technical approach. The research SenseTime has accumulated over the past year or two also gives its team a chance to produce results that differ from those of other companies.

Beyond technical research, SenseTime also adjusted its team organization ahead of time.

Recent reports say that companies including Tencent and Alibaba have begun merging their language and multimodal teams. SenseTime completed a similar adjustment as early as mid-2024.

At that time, SenseTime had already begun having the same model handle both language and multimodal tasks.

Later, when advancing research into unified understanding and generation, dozens of researchers from different departments could work together directly, without any further adjustment to the organizational structure.

🚥

From adjusting its teams ahead of time to continually investing in frontier research, what underlies these decisions is SenseTime's long-term judgment about the direction of multimodality.

But nobody yet knows how far this direction will ultimately lead. Everyone is finding their way through the fog.

Why call it fog?

Because even today, the AI industry has still not figured out exactly how to take the next step toward higher intelligence: where to go, which stretches to move through faster, and which to approach more cautiously.

Language models are still advancing rapidly, with continual breakthroughs in coding, mathematics, and reasoning. But as AI begins to engage with the real physical world, vision, spatial perception, and multimodality are becoming increasingly important again.

How exactly should these capabilities be combined? How much can multimodality improve intelligence? No one can give a definite answer at present.

The six judgments above are not intended to prove that SenseTime has already found the right answer.

They acknowledge the advantages of language models without denying the technical challenges facing multimodality. SenseTime has firmly chosen the path of multimodal general intelligence because it believes there are still many questions beyond language that are worth studying, and that these questions may determine the next stage in the development of intelligence.

Six judgments may not point to a single correct direction.

But for China's AI industry, which is still searching for its next step, besides continuing along paths that others have already established, there also need to be people willing to explore questions that remain unanswered.

Article illustration

This article comes from "Crossing," by "GaKi."

Related topics

After reading

Join the AINRK community

Read how to join the community and connect with people working in AI.

Community information (opens in a new tab)