VLM memory survey organizes research by time, location, content and operations

Researchers review more than 300 studies and nearly 70 benchmarks across video, embodied intelligence, multimodal retrieval and personal assistants, discussing compression, evidence traceability and the risks of outdated information.
Researchers from eight institutions, including Duke, Google, USC, UPenn and UNC, have jointly released a systematic survey of memory mechanisms in vision-language models (VLMs). Using a new four-dimensional taxonomy, it organizes more than 300 studies and nearly 70 benchmarks across eight application areas, including long-video understanding and generation, multimodal RAG, embodied navigation and action, and personalized assistants.

- Paper: Memory in Vision-Language Models: Taxonomy, Mechanisms, and Applications
- Authors: SHAO-JUN XIA, YIZHUO HE, YUNER ZHANG, YIFAN JIANG, XIAOYANG CHEN, LIANGXI LIU, CHENWEN LUO, JINBAO WANG
- Contributors: YIFAN JIANG, YIZHUO HE, YUNER ZHANG, SHAO-JUN XIA
- Project: https://github.com/Xzcv-hub/Awesome-Memory-in-VLM
- Paper: https://www.preprints.org/manuscript/202607.1539
1. Why VLMs need memory
VLMs have advanced rapidly in image captioning, visual question answering, video understanding, multimodal conversation and embodied intelligence, expanding toward more complex, large-scale, long-horizon tasks and real applications. These require continually accumulating and recalling past information over longer periods and more interactions. Enlarging context windows alone often cannot meet the need. Researchers have therefore introduced external memory stores, hierarchical memory, long- and short-term memory, and other mechanisms.
However, existing designs mostly start from specific task scenarios and architectures, and the implementation and optimization of memory also tend to focus on task-related dimensions. Their terminology and designs differ, and there is no unified perspective for understanding and comparing these methods. On the other hand, most existing surveys of memory focus on LLMs, while their coverage of VLMs includes only specific forms of tasks and therefore has certain limitations.
In this survey, the authors propose a new 4D taxonomy to systematically characterize memory mechanisms in VLMs. Based on this framework, they review more than 300 related studies, summarize eight typical forms of memory, compile nearly 70 evaluation benchmarks, analyze the role of memory in eight application categories such as long-video understanding, embodied intelligence and personalized assistants, and discuss current challenges and future directions.

Figure 1 | Timeline of VLM-memory research. Representative work and its development are shown across When (temporal scope), Where (storage location), What (memory content) and How (memory operations), with major contemporary foundation-model milestones in the center.
2. Core framework: the 4D memory taxonomy

Figure 2 | VLM memory's 4D taxonomy: When (temporal scope), Where (storage location), What (memory content) and How (memory operations).
The authors propose expanding the view of memory from a mere information-storage unit to its entire lifecycle. Relevant information is first observed in multimodal tasks and written into memory, repeatedly refined and organized over time, and retrieved and used when related tasks are being handled. Based on this perspective, the authors propose a 4D taxonomy that systematically characterizes memory mechanisms in VLMs along four dimensions: When (time horizon), Where (storage location), What (memory content) and How (memory operations).
1) When: how long memory lasts. This dimension concerns how long memory exists. The shortest-lived transient memory is used for a single inference, while episodic memory is suited to multi-turn conversations and multi-round interactions. Both kinds of memory are usually retained only during the current inference or task interaction. The longest-lived persistent memory, however, remains after a session ends and is used for subsequent interactions.
2) Where: where memory is stored. Storage can be internal or external. Internal memory resides in parameters, hidden states or KV caches, enabling fast access closely integrated with reasoning but making inspection and revision difficult. External memory uses separate databases, knowledge graphs or experience repositories, facilitating updates and review but depending on retrieval quality.
3) What: the form memory takes. This dimension concerns the form in which memory stores the original relevant information. Apart from keeping raw information fragments directly, memory can also be organized into structured dialogue and trajectory records or abstract semantic knowledge, or be compressed into latent representations inside the model.
4) How: how memory is operated on. This dimension concerns the operations memory undergoes throughout its lifecycle, mainly in four categories. Writing determines which new information is written into memory and in what form. Consolidation organizes and refines existing memories. Retrieval finds memories relevant to the current task to help complete it. Updating and forgetting update and clear information during ongoing interaction to ensure that the information remains timely.
3. Eight typical memory forms

Figure 3 | Eight typical memory forms positioned within the 4D framework.
The four dimensions can be discussed separately at a conceptual level, but they are often bound together in real systems. Choosing how long memory is retained usually also determines where it is best stored and in what form. On this basis, the authors summarize eight typical forms of memory (Figure 3), each corresponding to a typical combination of the four dimensions. For example, long-context memory retains tokens or their compressed representations through the context window and is usually transient, internal memory. Lifelong-agent memory, by contrast, needs to accumulate experience continuously across multiple sessions and is persistent memory. Accordingly, all four categories of operations—writing, consolidation, retrieval, and updating and forgetting—are considered in its design.
These combinations should be understood as common design tendencies rather than fixed rules. For example, although transient intermediate states cannot be retained after inference ends, their contents can be organized by additional components into episodic or persistent memory and used to assist subsequent tasks. The authors hope the relationships reflected in these combinations can not only help researchers clarify the memory-implementation preferences in existing work, but also point to design spaces that have not yet been fully explored.
4. Memory through applications: eight typical scenarios

Figure 4 | Memory requirements and preferences across eight application scenarios.
The eight forms of memory answer the question “How can memory be implemented?” In actual applications, however, which form to choose and how to combine forms depend to a large extent on the task itself. From an application perspective, the survey further compiles nearly 70 existing VLM memory benchmarks and explores memory implementations and the requirements prioritized across eight categories of application scenarios:
1)Long-video understanding and generation: because the model's context cannot accommodate vast numbers of video frames, memory implementations mainly focus on compressing visual information and selectively retaining task-relevant visual information.
2)Image reasoning and description: models may forget important information during extended reasoning, so memory emphasizes key visual information.
3)Continual learning: the core of these tasks is for the model to retain its previous capabilities while learning new knowledge and skills. Memory usually takes the form of protecting the model's original parameters and representation space.
4)Embodied navigation and action: memory mainly records trajectories, task progress and environmental spatial structure.
5)Multimodal RAG: these tasks mainly examine the construction of external knowledge bases and document images, as well as the efficiency and accuracy of retrieving relevant information according to the task.
6)Egocentric vision: these tasks involve long-term information spanning days or even weeks, and organizing memory hierarchically is often an important solution.
7)Personalized assistants: user-specific requirements and preferences must survive across sessions, favoring readily updatable external memory.
8)Virtual-environment interaction: these tasks examine whether an agent can distill past interaction experience into reusable knowledge and skills, and retrieve and refer to them when encountering new tasks.
5. Challenges and future directions
At the end of the survey, the authors outline the challenges that memory mechanisms in VLMs still face and future directions worth exploring. First, a longer context does not mean effective memory. Even if a model receives more information, it may still miss a brief but crucial event. Second, the importance of information often depends on what will be asked later. This delayed relevance makes it difficult to decide how to compress and retain related memories, and existing methods often preserve high-level semantic information while losing event order and visual detail. In addition, retrieved memories may themselves be irrelevant, outdated or even wrong. Yet models often lack the ability to verify those memories and can give seemingly reasonable answers even when relying on incorrect information, making the errors difficult to notice. Finally, existing benchmarks mostly test only a single memory capability, while errors in real task scenarios often involve multiple stages. Tests of individual capabilities cannot fully reflect the performance of memory systems in deployment.
To address these issues, the authors propose several directions worth exploring: draw on human cognition to build hierarchical memory structures, let systems adaptively choose storage methods according to information type, let models learn for themselves to judge what is worth remembering, and make every memory traceable to its original visual evidence. At the same time, evaluating memory systems should not focus only on accuracy. Whether users can view, modify and delete their own memories should also be considered from the beginning of the design process.
This article comes from Synced. Author: Synced.
After reading
Join the AINRK community
Read how to join the community and connect with people working in AI.




