Duo v1 explores reciprocal world models with full-duplex audiovisual interaction

An Ant team and Westlake Xinchen present Duo v1, combining intent decisions with audiovisual generation for continuous interaction, and release models and code. The team reports 30 cases and 180 human checks; the work remains an early research demonstration.
Imagine picking up a bottle of water in the real world and holding it toward a screen. A moment later, the virtual character on the screen reaches out and takes it.

More intriguingly, the virtual scenery can change with the conversation. As you discuss Hangzhou's West Lake with the character, a bright daytime setting gradually shifts to West Lake at night.

It sounds like science fiction, but a research team has already demonstrated it.
Recently, an Ant team and Westlake Xinchen proposed a new world-model paradigm, the Reciprocal World Model (RWM), and introduced its first implementation: the native world model Duo v1.
Duo v1 treats intelligence and physical reality as equally important foundations of a world. It aims to generate a world inhabited by lively intelligent agents and connect it to reality through real-time full-duplex interaction: responding while the other participant is still speaking or acting, proactively interrupting, or choosing to remain silent.

To do so, the team proposes two core capabilities: Intent understands the interaction, reasons about it and decides how to participate; Realization turns those intentions into coherent audiovisual behavior.
Users can also show the model real objects, change virtual scenes, guide an AI assistant or participate in audiovisual games, all without mouse or keyboard controls.
- arXiv: https://arxiv.org/abs/2611.15131
- Homepage: https://duo-v1.github.io/
- Github: https://github.com/inclusionAI/Duo
- Hugging Face Model: https://huggingface.co/collections/inclusionAI/duo-v1
- ModelScope: https://www.modelscope.cn/collections/inclusionAI/Duo-v1
From simulating a physical world to letting intelligent entities participate
To understand Duo v1, start with the team's assessment of existing world models.
Mainstream world-model research currently focuses on simulating the physical world. Models observe past frames and control signals to predict how future scenes will evolve. Users change the environment through keyboards, mice, controllers or text instructions, while the model generates the resulting images.
In this arrangement, users hold the initiative. Characters mainly exist to be observed or controlled, lacking the ability to independently perceive users, understand the interaction, decide how to respond or even create together with them.
In other words, generating a world is not the hardest part; making its characters truly come alive is.
Based on this assessment, the team proposed the Reciprocal World Model (RWM) and released its implementation, Duo v1.
Reciprocity is the key. Traditional interaction typically begins with a user instruction, followed by a wait for the model's result. RWM aims to make both sides active participants, continually perceiving one another's audio and video and reacting to events in real time.
While the user speaks, the virtual entity can decide whether to respond or even interject before the user finishes. When the user shows a real object, it can use visual information to understand the situation.
This reciprocal real-time interaction also distinguishes RWM from other common technical approaches.
Large language models primarily understand and generate language; traditional world models simulate environments and predict future states; and multimodal-agent or digital-human systems typically combine components such as ASR, LLMs, TTS and vision models to enable interaction.
Duo v1 explores a different implementation. Its two cooperating modules, Intent and Realization, integrate interaction decisions and audiovisual generation in one architecture.
A reciprocal world model thus incorporates intelligent entities' perception, cognition and behavior into the world model itself, forming a continuous two-way connection between the virtual world and real users.
From intent decisions to audiovisual generation: Duo v1's architecture
Duo v1 uses a modular design:
- Intent module: understands the interaction, reasons about user intent and decides when to respond and what to say.
- Realization module: translates those intentions into audiovisual behavior, generating speech, facial expressions and on-screen movement.

The modules communicate through hidden states output by Intent. These carry semantic features formed during interaction reasoning and supply Realization with the conditions needed to generate audio and video.
Intent: learning when to speak
Traditional audiovisual-generation systems typically use a language model as a conditioning encoder, passing interpreted text to a diffusion model to condition subsequent audiovisual generation.
Duo v1 gives Intent a broader responsibility: continuously understanding the ongoing interaction and independently deciding whether to participate.
Within each time chunk, Intent makes autonomous real-time decisions from user input and interaction history, while generating the corresponding language content.
- <speak>: begin speaking and generate the corresponding text.
- <wait>: generate no new assistant text in the current chunk and continue waiting.

For example, while a user is still speaking, the model can wait; when a question is asked or a suitable moment arises, it can proactively begin speaking. Importantly, <wait > does not mean the entire system stops producing output. Previously generated speech may continue playing, and Realization can continue the preceding audiovisual-generation process.
Intent passes more than its final generated text to Realization: it also outputs the corresponding hidden states, providing richer semantic conditioning.
To support real-time response generation, the team also encodes audio and video into compact latent representations for subsequent audiovisual generation.
Realization: turning intentions into coherent audiovisual behavior
Realization, Duo v1's core audiovisual-generation module, uses a Causal Audiovisual Diffusion Transformer.
The model starts from random noise and progressively generates audiovisual latent variables, conditioned on intent representations, reference images, currently available user input, and previously generated assistant audio and video content, among other inputs. The previously generated content serves as historical information, helping the model maintain continuity between different response chunks.
Intent conditioning: a linear projection maps Intent's hidden states to Realization's hidden dimension, then a Refiner Network generates text-conditioning embeddings.
Media conditioning: user audio and video are processed by audio-VAE and video-VAE encoders respectively to produce conditioning latents.
Reference images pass through the video-VAE encoder to produce reference latents. These latents are then mapped into the corresponding input embeddings of the joint Diffusion Transformer (DiT).
Previously generated assistant audio and video latents also enter the model as audiovisual history.
Together, these conditions supply four kinds of information for the current response chunk: real-time observations of the user, appearance features from reference images, semantic guidance and the model's own generated interaction history.
The model uses only user input obtained before chunk k begins; it cannot access subsequent information that has not yet occurred.
To maintain temporal consistency across modalities, the team introduces three-dimensional rotary positional embeddings (3D-RoPE), representing temporal and spatial relationships among audio, video and text tokens in a unified way. Audio and video align by actual playback time, while different spatial position encodings distinguish the left and right audio channels.
Duo v1 must also ensure strict temporal alignment across modalities.
The team maps user speech, video frames, Intent decisions and Realization generation onto the same timeline, organizing interaction into chunks of roughly 0.7 seconds. Specially designed attention masks in different modules control information access within and between chunks.
When generating the current chunk, the model can therefore use only previously obtained user input and interaction history, not future frames or user behavior that has not happened. Realization likewise generates audio and video only from currently available intent features, references and historical content.
Through joint generation, temporal alignment and causal constraints, Duo v1 combines user perception, intent understanding and audiovisual expression in one generation process, coordinating speech, expressions and on-screen movement around the same interaction intent.
From active decisions to autonomous generation: training Duo v1's interaction capabilities
Duo v1's training has three stages.

Stage one: supervised fine-tuning of Intent.
Training data are aligned by chunk. At each chunk's start, the model predicts what to output using global instructions, previously available user video, ASR transcripts and language content the assistant has already generated.
The model must learn two basic actions:
<speak > means begin speaking and generate new text; <wait> means introduce no new assistant text in the current chunk. Previously generated speech can continue playing even when it chooses to wait.
During training, the team found that most chunks produce no new assistant text. Training directly on that original distribution would make the model overly inclined to wait.
The team therefore designed Action-aware Rebalancing. Samples are categorized using information such as whether the user is speaking and whether the assistant's response begins or continues a turn. Subsampling and loss weighting then adjust the training proportions of different categories.
To adapt the model to imperfect interaction histories, the team also introduced History Perturbations: truncating or repeating historical text, or replacing a normal conversation opening with < wait>, to simulate incomplete prior responses, text deviations or missed opportunities to respond.
Stage two: training Realization for joint audio/video generation.
With Intent frozen, this stage extracts intent hidden states from recorded ground-truth assistant text to condition Realization semantically. User audio/video, character reference images and previously recorded assistant audiovisual history also serve as generation conditions.
Realization uses a causal audiovisual diffusion Transformer and flow matching to learn joint generation of audio and video latents.
Half the training uses a causal view, restricting information access to the order of real interaction. The other half uses a bidirectional view, letting the model learn audiovisual generation with fuller context.
The two training approaches each account for half the process and jointly optimize Realization.
Through this stage, Realization learns to jointly generate continuous audiovisual content from Intent's semantic features, user input and existing audiovisual history.
Stage three: Dual Rollout Distillation to narrow the training–inference gap.
During Realization training, intent features come from recorded ground-truth assistant text, and audiovisual history also comes from real data. During actual inference, however, the model must generate all of this information itself.
Relying on real recordings during training and the model's own generation during inference creates different input distributions. As interaction continues, this mismatch can affect subsequent generation stability.
The team therefore proposed Dual Rollout Distillation.

First, the frozen Intent module autoregressively generates the assistant's full text response for the current turn. This is called Language Rollout.
Realization then continues generating audiovisual content conditioned on the intent hidden states corresponding to that text and its own previously synthesized audiovisual history. This is Media Rollout.
The model thus encounters input conditions closer to real inference during training itself.
Finally, through Distribution Matching Distillation, the team trains on interaction trajectories it generates autonomously, further distilling the generation process into a Four-step Streaming Generator.
Real-time interaction also needs a streaming inference system
The team specially designed an Orchestrated Streaming Pipeline to coordinate scheduling and synchronization of Intent, Realization and the VAE for low-latency interaction.

The process operates chunk by chunk. At 24 FPS, each chunk contains 17 video frames, corresponding to roughly 708 milliseconds of sampling time. The frontend device continuously captures the user's audio and video and sends the data into the downstream processing pipeline. The model then carries out intent reasoning, joint generation of audio and video latent variables, and streaming decoding. Throughout the generation of a response, user input continues to arrive.
One Intent inference takes 90–160 milliseconds. Realization generates 720p audiovisual latents in 200–450 milliseconds, while transmission, display and other overhead take about 40 milliseconds. Together, these total roughly 330–650 milliseconds.
To reduce waiting, the team further optimized VAE decoding.
The original VAE decodes five latent frames into 17 video frames and requires two latent frames of lookahead. This dependency adds waiting at chunk boundaries. Duo v1 instead uses causal incremental decoding. Once a chunk is generated, its video latents can be decoded individually, with each latent frame corresponding to four video frames; temporal alignment then restores the original 17-frame playback cadence.
Each latent frame takes about 30 milliseconds to decode. At 24 FPS, the shortest playback interval is approximately 41.7 milliseconds. Provided generated latents arrive on time, decoding can keep pace with video playback.
The team also designed unified KV-cache management. Intent removes historical cache through a sliding window, while Realization resets media history at configured intervals.
For parallel computation, Intent uses four-way tensor parallelism (TP). Realization chooses an appropriate configuration between tensor parallelism and sequence parallelism (SP) according to compute speed and spare GPU memory.
The team's continuous-output target is a complete processing cycle no longer than 17/24 seconds per chunk—about 708 milliseconds—matching the input-sampling cycle.
The system's key is arranging input capture, intent reasoning, audiovisual generation and incremental decoding as a pipeline that runs in parallel. The model need not wait for the entire interaction turn to finish before responding, and users need not stop speaking while it completes all generation.
A 93.3% evaluation pass rate: Duo v1 explores a new interaction paradigm for world models
The team manually evaluated 30 real-time full-duplex interaction cases across six dimensions: content responsiveness, language consistency, speech intelligibility, visual identity stability, audiovisual synchronization and audiovisual emotional consistency.
In total, 168 of 180 checks passed, giving an overall pass rate of 93.3%. Language adherence, speech intelligibility and visual identity stability all passed every check, reaching 100%. This indicates the model could produce complete, clear speech and maintain character appearance in these checks. Audiovisual emotional consistency reached 96.7%, while lip–speech synchronization reached 90%.
Beyond basic conversation, the team demonstrated Duo v1 in additional interactive scenarios.
For example, users can resolve scheduling conflicts through conversation and ask the model to adjust plans. They can compare design samples and revise proposals together, or show a reference image and have the model change a clothing pattern accordingly.

The participants no longer have to wait for each other to finish speaking. Users can interject, interrupt or change topics at any time, while the virtual character perceives and responds concurrently.

Face a giant panda, China's national treasure, for a real-time interaction.
These examples show Duo v1's interaction design beginning to move beyond simple spoken questions and answers, combining language with visual observation, plan adjustment and collaborative action.
The next step for world models: letting agents truly participate in the world
What makes Duo v1 particularly interesting is its attempt to redefine the capabilities intelligent entities should possess inside a world model.
Traditional digital humans focus on character appearance, vocal expression and conversation. Users ask questions, and the digital human answers, using expressions, lip movements and gestures to enhance delivery. Interaction normally revolves around a predefined role, aiming to make the character look more real and express itself more naturally.
Reciprocal world models explore a broader space. By integrating perception, cognition, decisions and audiovisual behavior into the world model, they let virtual entities continuously observe users and their surroundings, independently decide when to participate and react to the interaction.
This opens more possible applications.
In games, NPCs could move beyond fixed dialogue and predetermined interactions, responding in real time to players' speech, actions and surroundings. Players could even interact directly with virtual characters through speech and real objects, turning one-way exploration into a process both sides actively shape.
For AI assistants, users could show real objects, images or environments so a virtual entity can join the discussion with visual context. It could help resolve scheduling conflicts, analyze design samples and keep revising plans in response to feedback. Interaction would extend beyond text boxes and voice interfaces, with the real environment itself becoming context for the model's understanding and actions.
For virtual-content creation, users could change scenes and adjust visual elements through natural language, or design and create alongside AI. A request to change the time of day could switch a virtual world from daylight to night; a reference image could guide changes to a character's clothing pattern. Language instructions, visual references and scene changes become parts of one continuous interaction.
Reciprocal world models also offer other directions to explore, including guiding AI assistants, changing virtual scenes and participating in audiovisual games without mouse or keyboard controls. Real objects can become inputs, virtual scenes can change with conversation, and AI entities can proactively join ongoing activities.
Duo v1 remains at an early exploratory stage, but offers a technical direction worth watching.
World models previously focused on generating a believable, coherent world. Reciprocal world models ask a further question: once intelligent entities inhabit that world, can they perceive reality, participate proactively and change the world together with people?
Future world models may be more than virtual spaces to watch and explore. They could become places where people and AI communicate, collaborate, design and create together.
This article comes from Synced. Author: Chenchen.
After reading
Join the AINRK community
Read how to join the community and connect with people working in AI.




