AI guides

CWM2.0 open-sources generative rendering with a 30-minute racing demo

By 山辉0 viewsSource: 机器之心
Generative rendering of a continuous racing scene

Teams from Ant, Westlake University and Zhejiang University use a structured Proxy to control scene state and Invoker-Render to generate visuals, releasing eight-GPU streaming inference. The long demo and 140 evaluations cover specified scenes, not arbitrary stable worlds.

Creating a world is not difficult. Making that world “obey” is the hard part.

Not long ago, Google DeepMind released Genie 4, showcasing the latest progress in interactive world models. Users can control characters with a keyboard or mouse to explore a world and change the environment through prompts. Many viewers declared that “game engines are dead.”

But actually trying it reveals a gap between the experience and expectations.

“Obeying” requires at least two capabilities:The first is control precision: the user's control commands must receive accurate responses. The second is temporal consistency: even while the world continues to change, objects' identities, appearances and states must remain stable.

Put simply, what should change must change, and what should stay the same must stay the same.

Now that AI can generate an explorable world, achieving precise control while keeping that world stable through continuous interaction has become a significant challenge for world models.

Recently, Ant Group, Westlake University and Zhejiang University jointly released Code World Model 2.0 (CWM 2.0). Its core visual engine, Invoker-Render, accepts structured Proxies and translates scene layouts, object motion and camera changes into continuous, high-fidelity imagery.

Article illustration

  • GitHub: https://github.com/inclusionAI/CodeWorldModel
  • Hugging Face: https://huggingface.co/inclusionAI/invoker-render-1.0
  • Project website: https://www.inclusion-ai.org/CodeWorldModel/

A racing demo from the team clearly illustrates the advantages of this control method.

The input is a low-poly racing Proxy, with the actual gamepad input shown at the lower right. As the tester moves the stick, the vehicle in the Proxy changes direction, and the white race car in the main image turns in sync.

Article illustration

More importantly, the team also showed a thirty-minute racing demo without cuts or resets, demonstrating the model’s ability to maintain visual stability and subject consistency during extended continuous generation.

How can a few simple wireframes drive a realistic world? And how can that world maintain consistent object appearances and scene structure throughout half an hour of continuous generation?

CWM 2.0’s answer is Invoker-Render.

This generative rendering engine understands Proxies at different levels of detail, from simple geometry to detailed skeletons, enabling precise control of scenes, objects and cameras. Its ProForcing training method also improves the stability of long-running streaming generation.

Code controls the world; the generative model handles the visuals

One mainstream approach to interactive world models currently uses end-to-end generation: the user supplies an action, and the model directly predicts and generates subsequent frames, without requiring a separately built, complex world program.

As interaction grows more complex, however, the model must handle two jobs simultaneously: maintaining object states, motion trajectories and interaction rules like a physics engine, while synthesizing realistic materials, textures and lighting like a graphics renderer. This design makes precise control and changes to world rules more difficult.

Another approach is to separate world-state management from visual generation.

In this architecture, code defines objects, motion rules and camera paths, converts that information into structured Proxies, and passes them to the generative model for visualization. Code provides explicit, editable control conditions; the model fills in materials, lighting and local details. This divides responsibility between world state and visual appearance.

In CWM 1.0, the team had already tried assigning world-state management to code and visual presentation to a generative model. CWM 2.0 builds on that approach to further improve generative rendering.

On the MiniMax-H3 foundation, the team built two complementary Invoker-Render models sharing the same input interface: structural Proxies, appearance-reference images and text conditions.

One is a bidirectional creation model, which reads the complete Proxy sequence and uses both preceding and subsequent context to generate frames. It suits offline video creation and complex camera-movement adjustments. A creator can, for example, plan an entire camera movement in advance and have the model generate corresponding frames from the structural conditions.

The other is a causal, four-step streaming model. It cannot see actions that have not yet occurred. Instead, it receives new Proxies chunk by chunk and combines them with previously generated frames to keep producing subsequent content. This approach is better suited to real-time interaction.

To make this division of work effective, the downstream renderer still has to solve two problems: understanding structural inputs at different resolutions and maintaining visual stability during continuous generation.

From a few wireframes to detailed skeletons,

how can world models achieve precise control?

A Proxy is the key link between the upstream world program and the downstream generative model. It can record scene layouts, object positions, orientations, motion trajectories and camera changes without fully specifying the final image’s materials, textures and lighting.

The problem is that different interactive scenarios require different structural information.

In a racing scene, for example, if the only requirement is to control a car driving along a road, a few simple 3D boxes can describe its position and orientation. Such a Proxy is inexpensive to construct and lightweight to transmit.

But in a character-combat scene, knowing only where a character moves as a whole is insufficient. The model also needs to know how the arms, legs and other body joints move to reproduce the corresponding action accurately.

How can the same model understand both coarse structures and detailed constraints?

Invoker-Render’s solution is to unify structural control at different granularities in the same model.

Five Proxy types, one unified generative model

Specifically, Invoker-Render supports five structural representations:

  • Semantic Map
  • Sparse 3D Boxes
  • Subject Mask
  • Shaded Geometry
  • Boxes + Skeletons

Article illustration

The five Proxy representations supported by Invoker-Render, with examples of coarse-to-fine geometric control in the same scene.

Besides supporting different types of Proxies, Invoker-Render can also adapt to different levels of geometric detail within the same representation.

Given a simple human structure, the model can fill in local actions such as stepping and arm swinging. With more detailed skeletal inputs, it can follow specific joint poses for tighter motion control.

To train the model to understand structural inputs at different resolutions, the team built approximately 200 hours of aligned, multi-granularity data. About 150 hours came from self-collected 3D interaction and driving sequences, while the remaining roughly fifty hours came from public datasets including OmniWorld-Game, Magpie, CARLA and Virtual KITTI 2.

Through scene reconstruction and object-motion recovery, the team generated Proxies at different resolutions from the same scene and paired them with corresponding real videos, teaching the model to generate images under different structural constraints.

The same event, different camera movements, different images

Beyond objects and actions, the camera itself is an important component that a world model must control.

In a series of coastal-tunnel chase demos, the same scene and event correspond to different camera paths. From the input Proxies, the model generates continuous frames from each viewpoint.

Article illustration

Article illustration

Different camera paths in the same scene: Orbit and Crane Up shots demonstrate the model’s control over camera movement.

A Proxy can therefore describe not only scenes and objects but also explicitly define subject motion and camera paths, providing more precise control over shots, actions and scene layouts.

Thirty minutes of continuous generation:

why did the race car’s appearance stay consistent?

If multi-granularity Proxies address “how to control,” ProForcing addresses “how to keep generating.”

Real-time streaming generation differs importantly from ordinary video generation: the model cannot see the entire video in advance. It must continually receive new inputs and generate subsequent frames chunk by chunk.

Once each chunk has finished generating, it becomes historical context for the next chunk. A problem follows: if earlier images have already introduced some errors, subsequent generation may continue to inherit those problems.

For example, an initially generated red car may have only a slight color shift. But as the model repeatedly reads its own generated history, that change may accumulate until the car becomes another color. Textures, lighting and object structures can change in similar ways.

To solve this, the model must learn not only to generate quickly but also to preserve visual quality when its own generated history contains errors.

ProForcing: dividing long-running generation into three training stages

The team therefore proposed a progressive training method called Progressive Forcing (ProForcing), separating causal adaptation, few-step distillation and long-sequence self-generated training into three stages.

Stage one combines Teacher Forcing (TF) and Resampling Forcing (RF): the former uses real video history as context, while the latter introduces some reconstruction error so the model gradually adapts to imperfect historical information.

Stage two uses RF-DMD to distill generation into four steps, combining GAN adversarial supervision to address the loss of detail and excessive smoothing that can occur with few-step generation.

Stage three uses SF-DMD (Self-Forcing DMD) to continue training the model on its own continuously generated history. The team also carries the student model, Fake Score model, discriminator, corresponding optimizer states and EMA weights into this stage to maintain training stability.

Article illustration

Training workflows for different streaming-generation models. Through TF→RF, RF-DMD+GAN and SF-DMD+GAN, ProForcing progressively performs causal adaptation, few-step generation distillation and long-sequence self-generated training.

A thirty-minute continuous shot demonstrates long-term stability

Returning to the opening racing demo, the vehicle keeps driving along the track while its body appearance, road structure and environmental lighting remain fairly consistent over extended generation.

The technical report also presents comparisons for two racing sequences of about five minutes each, lasting 298 and 299 seconds. In these sequences, Invoker-Render continuously follows the input trajectories while preserving the vehicle’s appearance.

Article illustration

Keyframe comparisons for two long racing sequences, with generation durations of 298 and 299 seconds.

The team’s separate full racing demo extends continuous generation to thirty minutes.

Article illustration

From a few seconds of generated demonstration to several minutes of continuous operation and then a half-hour uninterrupted shot, CWM 2.0 demonstrates the ability to maintain scene structure and subject appearance during long-running streaming generation.

From model to inference system:

how can a “generative graphics card” keep running?

Beyond model training, real-time streaming generation needs an inference system capable of sustained operation.

Even if a model can generate a video chunk in four steps, overall efficiency remains constrained if every inference repeatedly processes large amounts of history and encoding, generation and decoding must wait for one another in sequence.

CWM 2.0 releases not only the models but, alongside the report, a streaming inference engine for a single eight-GPU machine, built on SGLang and SGLang-Omni.

The team designed several engineering optimizations for continuous inference.

First, a structured block-causal mask reduces backbone forward calls per video chunk from six to four, cutting call overhead by about one-third.

Beyond four-step denoising, conventional streaming inference separately commits the previous chunk’s historical context and prefills the next chunk’s Proxy conditions. With a structured block-causal mask, the team fuses history commitment, condition prefilling and the first denoising step into a single forward pass. The remaining three passes complete subsequent denoising.

Second, cache reuse.

Resident appearance-reference anchors, persistent KV buffers and a zero-copy sliding window allow key historical information to be reused continuously in GPU memory. They also manage the growing history window, reducing repeated computation and data movement.

Finally, eight-GPU pipeline scheduling.

Proxy encoding, video generation and video decoding no longer wait in an entirely serial sequence. While the model generates the current chunk, the system can prepare the next Proxy and decode the previous output in parallel. Pipeline scheduling overlaps the different computational stages and reduces waiting.

Article illustration

CWM 2.0’s eight-GPU streaming inference architecture. KV Cache reuses historical context, while pipeline scheduling allows Proxy encoding, model generation and video decoding to run partly in parallel.

The accompanying repository also provides a WebSocket session service, a file-inference client and a browser front end supporting gamepad interaction.

This release therefore offers more than model weights. Developers also receive a matching streaming inference implementation and interaction tools, further lowering the barrier to trying and deploying the technology.

140 streaming evaluation sets: first on all four metrics

The team designed 140 held-out test sequences to compare Invoker-Render with three recent streaming rendering models.

These tests cover different 3D environments, multi-subject interactions and Proxy representations. Evaluation uses Gemini 3.8 Flash for anonymous pairwise comparisons, then fits Elo scores through a Bradley–Terry model.

There are four dimensions: visual quality, structure and motion, reference-appearance preservation, and overall preference.

In these tests, Invoker-Render ranked first on all four metrics:

● Visual quality Elo: 1336.4

● Structure and motion Elo: 1309.0

● Reference-appearance preservation Elo: 1214.1

● Overall preference Elo: 1343.2

Against each of the three streaming rendering baselines, Invoker-Render achieved a higher overall preference, with win rates of 92.9%, 82.9% and 62.9%. Against the most competitive baseline, Other Work-3, its overall-preference Elo lead was 97.9 points.

Article illustration

In seven streaming-generation comparisons, other methods showed clear structural and appearance drift.

Article illustration

Streaming-generation comparisons on seven held-out sequences. Compared with other methods, Invoker-Render more closely matches the reference imagery in subject appearance, scene structure and motion.

For example, in an underwater scene, environments generated by other methods gradually drifted away from the original underwater reef and even turned into bright skies. A running deer lost its antlers, a winged character lost its wings, and an originally vivid red sports car gradually faded to cream. Some scenes also showed obvious deterioration in road textures, or even entire areas progressively turning black.

These changes show that streaming world rendering must maintain multiple constraints simultaneously: object motion must follow structural inputs, while subjects must preserve their original appearance over time.

World models enter a programmable era

When world models can not only generate but also allow precise control and continuous modification, their applications can extend beyond images for people to watch. They have the potential to become new spaces for creation and interaction.

In the Akina racing demo, users change the vehicle’s motion with a gamepad. The Proxy at the lower left changes accordingly, while the main image simultaneously shows the vehicle turning and the track environment changing. This offers a new approach to game prototyping, interactive virtual environments and similar applications. Developers can combine structured control with generative rendering rather than relying entirely on traditional graphics pipelines for every visual detail.

Film production is another promising direction. In a two-shot demo of one race car overtaking another, creators can adjust the camera path to generate different viewpoints of the same event. This also provides a new tool for shot previsualization and storyboard design.

Article illustration

From CWM 1.0’s Coding Agent as World Brain concept to CWM 2.0’s further improvements in generative rendering, this approach is gradually filling in the key steps from world building and rule control to real-time visualization.

In the future, a world model may be more than a system that generates a video from a prompt. It could be a world whose rules are defined through code, whose state changes through interaction and whose visuals are continuously rendered by a generative model. Making generated worlds truly “obey” is gradually moving from a technical idea toward reality.

This article was originally published by Synced. Author: Shanhui.

Related topics

After reading

Join the AINRK community

Read how to join the community and connect with people working in AI.

Community information (opens in a new tab)