AI guides

miHoYo unveils Gamer VLA research for long-horizon memory and real-time game control

By 量子位0 viewsSource: 量子位
Gamer VLA research by miHoYo’s Honkai AI R&D team

miHoYo’s Honkai AI R&D team presents Gamer, generating keyboard and mouse actions from screens, instructions and memory. The team reports over 95% autonomous runtime across five Star Rail story chapters and about 50 hours; this is not a task success rate.

miHoYo, are you getting into researching “official game scripts” yourselves? (doge)

In the latest study from miHoYo’s Honkai AI R&D team, Gamer—

Would you believe it? AI is playing Honkai: Star Rail by itself.

Hand the keyboard and mouse to AI, and it can jump straight into a fight with Cocolia.

Article illustration

It can also operate environmental mechanisms on its own: walk up to a Bubble Pinball machine, activate it, launch into another area, then keep running after landing.

Article illustration

Following an NPC across the map works too. Firefly leads and the AI follows, climbing stairs and turning corners all the way to the square.

Article illustration

Goodness—AI has learned boss fights, mechanisms and following people across maps.

Might the top-up page eventually offer an “AI grinding service” option?

(Note: Honkai: Star Rail is miHoYo’s turn-based role-playing game about a galactic adventure. Players ride the Astral Express to explore different worlds, advance the story through navigation, dialogue and puzzles, and assemble teams for turn-based combat.)

Let VLA do the grinding, without the stress

Back to the research: miHoYo positions Gamer as a general-purpose vision-language-action (VLA) foundation model for long-horizon, real-time interaction.

Article illustration

The main version demonstrated playing Star Rail was further trained from the eight-billion-parameter Qwen3-VL-8B model.

From the game screen and task, it decides which keys to press, where to move the mouse and when to click, then actually executes those operations.

In other words, screen images provide visual input, task instructions specify what to do, and keyboard and mouse operations are its action outputs.

Compared with earlier game-playing AI, Gamer’s main feature is combining cross-game real-time control with long-horizon memory and planning in one VLA framework.

Article illustration

During play, the model makes decisions entirely from screen images, task instructions and existing memory, without reading internal game interfaces or underlying state.

As a task progresses, it dynamically records what it has just completed and what it intends to do next.

The paper also reports a concrete evaluation: across five main-story chapters and approximately 50 hours of long-horizon Star Rail testing, Gamer’s autonomous runtime proportion exceeded 95%.

Article illustration

To achieve this, the researchers designed a three-stage training process:

First learn controls from cross-game data, then add thinking and memory through task demonstrations, and finally enter the game to improve through trial, error and feedback.

The first stage is Continued Pre-Training (CPT), teaching a backbone that already understands images and text to operate games.

The team aligns recordings from different games with players’ inputs, teaching relationships between images and actions: how the scene changes when a character moves forward, or where the view turns when the mouse moves right.

These inputs are organized into a unified keyboard-and-mouse format so data from different games enters the same training process.

At execution time, Gamer reads a frame every 200 milliseconds and schedules six subsequent action chunks, each about 33 milliseconds, corresponding to a 30 Hz control cadence.

It is like taking one look, arranging a brief sequence of actions, then adjusting from the next image.

Next, in fine-tuning, the team assembled approximately 1,000 hours of high-quality Star Rail demonstrations, dividing goals into concrete actions, intermediate subgoals and complete tasks to teach the model to organize actions.

Two particularly useful engineering designs address inference latency and exploding context length in long-horizon tasks:

-The “sparse thinking” mechanism: On an established route, it can act directly; it reasons when the route fails, the goal changes or a new interaction prompt appears. — The memory-summary mechanism: Before old records leave the context, it notes progress, failed attempts, important locations and future plans, giving subsequent actions a basis to follow.

—“Sparse thinking”: On an established route, it can act directly; it reasons when the route fails, the goal changes or a new interaction prompt appears.

Finally, for online reinforcement learning, the team proposed TA-GRPO. It splits long-horizon workflows into memory-linked segments, compares different attempts at the corresponding stages, and updates the model using outcomes verified by the game environment.

Whether the model reached a destination or completed the current task can provide training feedback, with the training backend handling initialization and verification.

The accompanying Streaming-vLLM inference engine also reuses processed context and streams generated, verified action chunks early, allowing the game to move immediately and substantially reducing action latency.

Experimental tests

Cross-game experiments tested Gamer’s CPT version in Minecraft, the Doom-based ViZDoom, OmniGameArena and GameWorld.

In Minecraft, for example, it can turn the camera to lock onto a slime, chase it and swing a sword:

Article illustration

In the first-person shooter ViZDoom, it rapidly aims and fires repeated bursts at approaching enemies:

Article illustration

Across the four public benchmarks above, Gamer-8B-CPT averaged 29.5, versus a maximum of only 17.6 among the listed general-purpose comparison models.

Across 40 unseen tasks in seven minigame categories, the base model scored only 3.2%; CPT raised this to 15.0%, SFT to 39.2%, and reinforcement learning to 49.7%—

By comparison, the highest general-purpose model score in the same evaluation, GPT-6.1-Sol, was only 17.1%.

Article illustration

These results support the team’s central view: a unified keyboard-and-mouse interface combining cross-game learning, long-term memory and real-time control can enable AI to sustain tasks in complex games and improve through interaction.

The paper concludes that these capabilities could support automated testing, accelerate game iteration and enable new gameplay mechanisms, reducing development costs.

Games become AI’s training ground, and the trained AI can in turn help develop games.

About the team

Finally, let us meet the researchers behind the work.

The contributor list names eight co-first authors, five of whom worked on the project while interning at miHoYo:

Article illustration

Run Luo: A Computer Science PhD student at the National University of Singapore who has worked on GUI-R1 and OpenOmni. For Gamer, he worked on the three-stage training, data, infrastructure and evaluation.

Article illustration

Kuan Zhang: A PhD student at Tsinghua University’s School of Artificial Intelligence and co-first author of GameScaling and GameVerse, mainly responsible for continued pretraining, data and evaluation.

His personal website also notes that he is a Master-ranked League of Legends player.

Article illustration

Qi Deng: A member of miHoYo’s Honkai AI R&D team, responsible for reinforcement learning, training/inference infrastructure, data and evaluation.

Yukun Chen: A graduate student at the University of Chinese Academy of Sciences, co-first author of GameScaling and first author of the multimodal-reasoning study RuCL, mainly responsible for supervised fine-tuning, data and evaluation.

Zhihao Yang: A Computer Science PhD student at the University of Chinese Academy of Sciences, with an undergraduate degree from Tongji University, focusing on games and agents. He worked on reinforcement learning, data and evaluation.

Article illustration

Yufei Zhang: A member of miHoYo’s Honkai AI R&D team, mainly responsible for training and inference infrastructure.

Renrong Hu: A member of miHoYo’s Honkai AI R&D team, mainly responsible for benchmark construction and evaluation.

Yi Zhang: From the University of Science and Technology of China, he participated while interning at miHoYo, mainly working on benchmark construction and evaluation.

(Other contributors can be found on the paper’s website.)

More broadly, miHoYo’s Honkai AI R&D team spans AI NPCs, game agents and R&D automation, exploring large-model applications in new game experiences and industrialized development pipelines.

One of the research supervisors, Yinhe Zheng, leads the Honkai series’ AI NPC and Gameplay technology team.

Zheng previously presented Star Rail’s AI Pom-Pom and EchoX, an agent platform supporting internal R&D, at the Apsara Conference.

Academically, the team has also collaborated with universities on GameScaling and Video-Arena, studying scaling laws for cross-game continued pretraining and the limits of models’ understanding of dynamic game imagery, respectively.

References

[1]https://arxiv.org/html/2611.18397v1

[2]https://mihoyo-gamer.github.io/

This article originally appeared on the WeChat account “QbitAI,” written by QbitAI.

Related topics

After reading

Join the AINRK community

Read how to join the community and connect with people working in AI.

Community information (opens in a new tab)