ConvexLLM connects speech to language models through frozen text embeddings

Researchers at CUHK-Shenzhen and Microsoft propose C-Gate, mapping audio into convex combinations of frozen text embeddings. The paper reports VoiceBench gains against its listed open-model baselines, while audio understanding and transcription still involve trade-offs.
This research was conducted jointly by The Chinese University of Hong Kong, Shenzhen, and Microsoft Research. Corresponding author Zhizheng Wu is an associate professor at CUHK-Shenzhen, a recipient of a national young-talent distinction, and founder of Amphion Technology. Shujie Liu is a principal researcher at Microsoft Research, with a longstanding focus on natural language and speech processing. Ming-Hao Hsu, Yuxiang Wang and Yingda Shen are doctoral and master’s students at CUHK-Shenzhen. Yuxuan Hu and Jinyu Li are from Microsoft CoreAI. The proposed ConvexLLM uses an innovative C-Gate architecture to map speech information into the convex hull of a large language model’s existing text embeddings, allowing it to draw on its existing knowledge and reasoning abilities. On VoiceBench and other benchmarks, ConvexLLM achieved leading performance among similarly sized open models, providing a new technical approach to unified speech and language modeling.

- Paper title: ConvexLLM: Scaling Convex-Hull Speech Interfaces to General-Purpose Audio Intelligence
- Paper: https://arxiv.org/abs/2610.23539
- Authors: Ming-Hao Hsu, Shujie Liu, Yuxuan Hu, Yuxiang Wang, Yingda Shen, Jinyu Li, Zhizheng Wu
- Affiliations: School of Data Science, The Chinese University of Hong Kong, Shenzhen; Microsoft CoreAI; Microsoft Research Asia
Large models have already learned knowledge and reasoning. If we give them a pair of “ears,” can those capabilities carry over too?
Speech large models cannot avoid this question: a model may answer a written question well, yet perform significantly worse when the same question is spoken. The sound has already been encoded into vectors, and the language model is powerful enough. What is still missing between them?
A research team from CUHK-Shenzhen and Microsoft focused on the interface through which sound enters the language model. ConvexLLM constrains audio representations to the convex hull of the language model’s existing text embeddings, letting sound enter the model as continuous combinations of text vectors.

ConvexLLM demo
What is a “convex hull”? Imagine pins fixed to a tabletop. Stretch a rubber band around them from the outside and let it tighten. The enclosed region, including its boundary, is the convex hull of those points. Any weighted average of the points using nonnegative weights that sum to 100% will lie within that region. Text embeddings can also be viewed as points in a space, just with many more dimensions. ConvexLLM represents sound by mixing these embeddings.
This seemingly restrictive design delivered striking performance in speech-based knowledge and reasoning.
In the paper’s comparison of similarly sized open-weight models, ConvexLLM ranked first on the aggregate score across VoiceBench’s eleven tasks, with 75.36. It reached 90.33% on OpenBookQA and 80.40% on BBH, the latter 8.80 percentage points above the strongest baseline in the table.
The study brings a question into focus for the entire multimodal field: how much information about sound can the continuous space formed by text embeddings actually carry?

- Visit the ConvexLLM paper website: https://mhhsu1006.github.io/convexllm/
1. Giving sound coordinates the language model already knows
Text large models acquire extensive knowledge, reasoning patterns and instruction-following abilities during pretraining. A core aim of speech large models is to let users access those same capabilities simply by speaking.
After passing through an encoder, sound must be converted through an interface into language-model input. Common interfaces such as MLPs and Q-Formers can map freely in a continuous space. ConvexLLM chooses an explicit reference for this step: the language model’s own text embeddings.
Its core interface, C-Gate, represents each audio position as a weighted combination of multiple frozen token embeddings. The embeddings provide stable coordinates; the sound determines how they are combined. This design starts speech adaptation from representations the language model already possesses.
From this perspective, for speech and text expressing the same content, the difference in modality need not imply a difference in semantics. It may instead reflect different trajectories through a shared representation space. Text organizes information through discrete tokens, while speech can carry content and acoustic detail through continuous combinations of vectors that change over time. C-Gate provides a common coordinate system for exploring this idea.
Let sound use representations familiar to the large model to access the intelligence it has already learned.
2. The embeddings are fixed; their combinations are flexible
The most interesting aspect of “putting sound into text space” is continuous combination.
Think of the language model’s embeddings as a palette. The pigments are fixed, but changing their proportions produces a rich range of colors. C-Gate does something similar for sound: each audio position first forms a query, selects the sixteen best-matching token embeddings from the vocabulary, and then learns their mixing weights.
These weights are nonnegative and sum to one, so the mixed representation lies within the convex hull of the text embeddings. As the sound changes, the selected vectors and their proportions change with it. The representations at successive audio positions form a temporal trajectory within that convex hull. What remains fixed is the reference for the representation space; what changes is the path the sound traces through it and the information that path carries.

Figure 1 | C-Gate continuously mixes the selected top-16 frozen token embeddings, allowing sound to enter the language model.
This preserves an important expressive freedom: an audio representation can lie between multiple token embeddings. It need not correspond to a single discrete word or first produce a readable transcript. What the language model receives remains a sequence of continuous representations determined by sound.
The importance of continuous mixing is especially clear in the support-set-size experiment: in the paper’s 2B configuration, selecting just one token embedding per position produced an aggregate VoiceBench score of 22.30. Expanding to continuous combinations of sixteen token embeddings raised the aggregate score to 60.35. Freedom of combination makes this fixed embedding space more expressive.
Training also proceeds gradually around this interface: first the interface is trained, then the encoder is included, and finally the encoder, interface and decoder modules are adapted jointly. The text embedding table remains fixed throughout. The main model also uses distillation from a text teacher, training its speech-input response distribution to learn the language abilities elicited by the corresponding text input.
3. A score of 75.36: connecting sound to stronger knowledge and reasoning
The value of an interface design ultimately depends on how much capability it brings to speech.
On VoiceBench, which covers speech interaction, question answering, instruction following, reasoning and other tasks, ConvexLLM achieved an aggregate score of 75.36 across eleven tasks, above MiniCPM-o 4.5 at 73.04, Kimi-Audio at 70.26 and MiMo-Audio at 67.01.

Figure 2 | ConvexLLM leads the aggregate score across VoiceBench’s eleven tasks with 75.36.
The most striking results are in knowledge and reasoning. ConvexLLM achieved the highest scores in the table on all four tasks: OpenBookQA, MMSU, SpeechMMLU and BBH.

Figure 3 | All four knowledge and reasoning tasks exceed their respective strongest baselines, with an 8.80-percentage-point lead on BBH.
Specifically, BBH improved from the strongest baseline’s 71.60% to 80.40%, while OpenBookQA rose from 83.70% to 90.33%. These tasks require a model, after hearing a question, to draw on knowledge, analyze conditions and reason through an answer. The gains occur in precisely the abilities we most want speech large models to acquire: after understanding what was said, actually thinking the problem through.
For research into applications such as voice assistants, spoken-language learning, knowledge-based question answering and professional advice, this offers an appealing direction: connecting the speech interface more effectively to the language model’s existing reasoning capabilities.
4. The convex hull of text can also carry environmental sounds and music
ConvexLLM further applies the same interface to understanding speech, environmental sounds and music.
On MMAU Test-mini, overall accuracy was 62.50%, comprising 65.77% for environmental sounds, 60.78% for music and 60.96% for speech. It also performed speech recognition, achieving a 2.39% word error rate on LibriSpeech test-clean.
This gives “text space” a broader meaning: embeddings originally used for text can be continuously combined to carry the information required by multiple audio tasks. The vocabulary is finite, but combinations preserve a rich range of variation for audio.
Built on existing acoustic encoders and language models, ConvexLLM used an upper bound of 271,000 hours of audio during adaptation. A clearly structured interface was sufficient to support a complete research system spanning speech recognition, knowledge-based reasoning and non-speech audio understanding.
5. The interface itself deserves serious study as a variable
If you work on speech LLMs, your next question is probably how much difference the choice of interface itself can make.
Using Qwen3.5-2B, the team compared MLP, Q-Former and C-Gate interfaces, each with approximately 1.7 million parameters. All three used the same encoder and decoder and shared the Stage 3 data, training objectives and optimization settings. Evaluation followed 18,000 steps of Stage 3 training.

Figure 4 | With the same 2B backbone, C-Gate has the highest aggregate speech score and the smallest gap from the corresponding text input.
C-Gate’s aggregate speech score was 60.35, above the MLP’s 58.80 and the Q-Former’s 54.19. The aggregate score with text input was 63.20. Of the three interfaces, C-Gate also had the smallest aggregate-score gap between speech and the corresponding text.
Representation analysis provides a more direct clue. In another set of 2B diagnostic models, by the seventh decoder block, the centered cosine similarity between C-Gate’s speech and corresponding text representations had reached 0.69, compared with 0.45 for the MLP and 0.44 for the Q-Former. Relatively early in the model’s computation, speech begins following a representation path closer to that of text.
This gives interface research a concrete point of investigation: where does sound sit when it enters the language model, and how does that affect subsequent knowledge access and reasoning? ConvexLLM turns this question into an experimental design that can be implemented, compared and analyzed.
6. The next study could start with this interface
What most deserves further exploration in ConvexLLM is the simple, explicit starting point it provides for formulating research questions.
For speech large models, start with stronger capability transfer
Include C-Gate in bridge comparisons and study stronger speech instruction following, complex question answering and reasoning on the same backbone. The interface gives sound a stable textual reference. Further work can explore how distillation signals, data recipes and training stages interact to bring more language capabilities into speech.
For representation learning, ask directly what text space can hold
Content, emotion, speaker identity, temporal order and environmental events can each be studied through the same interface. Changing the number of mixed token embeddings, the degree of input compression or the backbone lets researchers observe which information enters the representations and which information the decoder actually uses. How rich a world of sound can finite token embeddings combine to represent? This is both a concrete question and a substantial one.
For multimodal adaptation, use it as a preliminary study
ConvexLLM’s interface concept offers a clear experimental starting point for further exploration: fix the embedding table, learn how to combine inputs, and compare different encoders, frozen or trainable decoders, and new modalities and tasks. It turns “what existing language intelligence can accommodate” into a set of design choices that can be tested incrementally.
For researchers, that value is direct: a clear structure, comparable results and many interesting questions left open. You can put it into your own experiments to investigate what additional information a change lets sound carry in, and what more it lets the large model use.
Text embeddings were originally the entry point through which language models recognized text. ConvexLLM shows that this entry point can also open to sound.
From this perspective, the boundary of text can become a new starting point for exploring audio intelligence.
This article was originally published by Synced. Author: Synced.
Related topics
After reading
Join the AINRK community
Read how to join the community and connect with people working in AI.




