Shishi introduces multimodal inference services combining cache reuse and token compression

Shishi reports average 2.48-times video-inference acceleration over Dense in launch tests, alongside GPU-memory scheduling, asset understanding and structured context for generation. Results apply to the demonstrated settings.
In AI video generation, faster inference can coincide with lower quality.
Reducing the number of computation steps can shorten generation time, but it may also cause errors to accumulate; retaining the full computation process can maintain generation quality, but entails higher compute and GPU memory costs.
As video moves from isolated trials to batch production, the question becomes how to balance quality, inference latency and resource costs.
To address this, Shishi Technology has launched its Multimodal Model Service, focusing its optimization efforts on accelerating inference for video generation, managing caches and GPU memory, compressing multimodal tokens, and connecting asset understanding with the generation pipeline.
Its components include:
- Adaptive acceleration based on historical cache residuals to reduce redundant video computation;
- Cache management combined with SGLang to optimize resources;
- Modality-adaptive token compression to reduce understanding costs;
- An omni-modal Context Engine converting understood assets into structured instructions executable by video models.
These act at different stages toward one goal:
better multimodal inference efficiency and compute utilization while maintaining quality and service stability.
Video acceleration: reduce redundancy while controlling accumulated error
Video generation models usually need multiple rounds of iterative computation to gradually generate video content from the initial information. During this process, there is information that can be reused between adjacent computation steps.
If the full computation is performed at every step, inference overhead will be high; yet if some steps are skipped outright or historical results are reused, cache errors may keep accumulating through the iterations and ultimately affect generation quality.
The key is therefore not simply fewer calculations, but deciding what can safely be reused and when recomputation is needed to control error.
Shishi uses an adaptive acceleration technique based on historical cache residuals. It uses historical computation information to assess the risks of reusing cached results and dynamically adjusts its computation and refresh strategies.
For parts of the computation that change little, the system reuses historical results to reduce repeated computation; for critical parts with a higher risk of error, it triggers recomputation and calibration to reduce the impact of cache errors on the subsequent generation process. Through dynamic refreshing and error calibration, the system reduces computational overhead while preserving the consistency of the video generation trajectory as far as possible.

△A visualization of Shishi’s cache-reuse acceleration versus Dense under identical experimental settings

Under the launch demonstration’s test conditions, the approach achieved average end-to-end acceleration of 2.48 times the Dense baseline while maintaining quality.
This means inference acceleration no longer relies entirely on reducing generation steps; instead, it uses finer-grained computation-reuse strategies to establish a dynamic balance between efficiency and quality.
Joint cache and GPU-memory optimization: more tasks per unit of compute
Video costs come from GPU-memory occupancy as well as computation.
When multiple video tasks run concurrently, the storage, reading and updating of cached data all affect how efficiently GPU memory is used. Even if an individual task is accelerated, additional resource burdens introduced by cache management may still limit the overall capacity to process tasks.

Thus, single-task acceleration and multitask resource use require joint optimization.
Shishi combines intelligent cache decisions with the SGLang inference framework. Through mechanisms such as cache reuse, chunked processing, compression and intelligent scheduling, it reduces repeated computation and unnecessary GPU memory overhead.
At the same time, the system introduces a risk-aware mechanism that dynamically adjusts caching strategies based on actual computation feedback, improving reuse efficiency while controlling the risk of errors. This design extends the scope of optimization from an individual inference computation to resource management.
For enterprise video production, generation speed, GPU memory usage and the capacity to run tasks concurrently jointly determine the actual output of compute resources. Reducing redundant computation and cache overhead can make room for greater concurrency and more flexible task deployment.
Modality-adaptive token compression: reducing audiovisual understanding cost
Understanding multimodal inputs also raises efficiency challenges.
Compared with text, video and audio contain a higher density of temporal information. When a model processes inputs such as consecutive video frames and sound segments, it needs to convert this content into a large number of tokens, increasing the computational and storage burden. However, information density differs across modalities and across segments. Some tokens contain key semantic information, while others have a high degree of redundancy.
Treating all tokens in the same way can easily create unnecessary computational overhead; directly applying uniform compression, however, may lose important information that affects the results of understanding.
Shishi proposes training-free, modality-adaptive token compression, using AudioCLIP-based modality awareness to identify useful information and adjust retention and compression.
The core idea is allocating resources by modality characteristics and information value rather than compressing every input proportionally.
It requires no extra training, reducing redundant tokens and understanding costs while retaining key semantics as far as possible.

In the WorldSense evaluation, this method achieved a 1.33-fold speedup. It also demonstrated good performance on multimodal understanding benchmarks including VideoMME and DailyOmni. This further extends multimodal inference optimization from video generation to asset understanding, providing a more efficient technical approach to processing complex audiovisual inputs.
An omni-modal Context Engine
Beyond efficiency, video creation must connect understood material to generation.
Real requests often combine character images, reference videos, audio and text.
They contain constraints on identity, action, voice, shots and temporal order, not just different data formats.
If complex assets and natural-language instructions are passed directly to a generation model, the relationships among the assets and the execution requirements may be difficult to express consistently.
Shishi therefore built an omni-modal Context Engine solution, structuring the information passed between understanding and video generation.
The system first understands the input assets and creative requirements, extracts characters, voices and the relevant constraints, and then organizes them into structured information containing shots, actions, dialogue and other elements for downstream video generation models to execute. For example, when a user provides character images and reference audio and requests a video of character dialogue, the system needs to connect the character information, audio content and shot arrangements, rather than simply generate a text prompt.
Through mechanisms such as asset binding, structured storyboarding, timeline validation and error correction, this solution enables information from different modalities to form relatively clear correspondences within the generation process.
The focus is converting complex inputs into clear, checkable and executable generation context.

At the service level, Shishi has further built a multimodal inference architecture covering API access, task orchestration, model routing, inference execution and result delivery. Through task-state management, concurrency control, retries after timeouts and recovery mechanisms, it supports the running and delivery of tasks in actual business operations.
From inference efficiency to a complete production chain
Once multimodal models enter production environments, generation quality, inference latency, GPU memory costs and task stability are no longer independent metrics.
The video generation side needs to control redundant computation and the accumulation of errors; the multimodal understanding side needs to reduce the processing of ineffective tokens; and the generation pipeline needs to ensure that asset information and creative constraints can be passed along effectively.
The service considers these optimizations together in one production chain.
From adaptive cache acceleration and GPU memory resource management, through modality-aware token compression and the structured connections provided by the Context Engine solution based on omni-modal models, to the inference service architecture, optimization progressively extends from the computation process of an individual model to the execution and delivery of multimodal tasks.
The key to applying AI video at scale is not just shortening the wait for a single generation, but continually completing more generation tasks that meet quality requirements within a given amount of compute resources.
Moving from “can generate” to effective production, Shishi plans continued optimization of multimodal efficiency, resource utilization and service stability to support practical video production.
This article was originally published by QbitAI. Author: Yunzhong.
Related topics
After reading
Join the AINRK community
Read how to join the community and connect with people working in AI.




