AI guides

EM²Mem organizes multimodal memory around events, linking people, actions and temporal evidence

By 机器之心1 viewsSource: 机器之心
EM²Mem event-centric multimodal memory linking people, actions and temporal evidence

A Zhejiang University and Lenovo team aligns modalities when building memory and tests long-video question answering and evidence localization. Queries are lighter, but upfront construction costs must be counted.

"Who just said they were planning to grow flowers?"

People can often answer by recalling the recent conversation and scene. But for an AI that has watched hours of video, the question is not simple. It might find flowers, several people's names and a discussion of planting, yet still be unable to determine who actually proposed the plan.

The clues were found, but their relationships were not remembered.

This reveals a key issue in long-term memory for multimodal agents: memory should do more than simply store past information. It should also understand the relationships among people, objects, actions and events within experiences.

Particularly as AI assistants begin accompanying users over the long term, memory's value extends beyond answering a particular question. It also needs to gradually understand who users are, what they have done, what they care about and their preferences, linking new experiences from every interaction with existing memories. Through continued use, it must keep accumulating, correcting and consolidating these into truly personalized memory. This is also an important foundation for agents' continual learning and recursive self-improvement (RSI).

To address this issue, a joint research team from Zhejiang University and Lenovo Group proposed EM²Mem (Event-Centric Multimodal Memory), which builds event-centric multimodal long-term memory. Instead of storing video frames, spoken conversations and text in isolation, it connects the people, objects, actions and temporal clues within the same event during memory construction. This lets AI not only find what happened, but also understand whom it concerns and how preceding and subsequent experiences connect.

The core idea is: organize experiences around events, consolidate memory through relationships, and support continual learning and personalization in multimodal agents through ongoing accumulation.

Article illustration

Figure 1 | EM²Mem builds event-centric multimodal long-term memory.

  • Paper: EM²Mem: Event-Centric Multimodal Memory for Large Language Models
  • Conference: EMNLP 2026 Findings
  • Paper link: https://arxiv.org/abs/2609.00551
  • Code integrated into: https://github.com/zjunlp/LightMem-Ego

01 | Why finding relevant content is not the same as remembering

Evidence in long videos is often fragmented and spread across modalities.

Someone's name may appear in conversation; seeing what they are holding requires the image; determining whether a task was completed may require combining several experiences before and afterward. Users' questions, however, are usually complete: "Who proposed this plan?" "What happened to this afterward?" "What do I usually do after arriving at the office?"

Many existing methods process video into descriptions, transcripts, keyframes or graph facts, store them separately, then retrieve relevant content when a question is asked. This finds clues, but the model must still work out on the spot whether they concern the same event, who relates to whom, and what came first.

The paper summarizes this bottleneck as retrieve-then-align: retrieve first, then align.

If an individual piece of evidence does not look relevant enough on its own, it may even be filtered out during retrieval, losing the opportunity to explain the question together with another clue.

EM²Mem reverses this order: align-then-retrieve—align first, then retrieve. It moves evidence organization into memory construction so subsequent question answering can work directly around events.

02 | Creating an event file for an experience

An event memory unit in EM²Mem can be understood as an event file with a temporal address.

The system first divides a long video into shorter base segments. The typical granularity used in the paper is 30 seconds. Each segment corresponds to an event anchor that records its identity and start and end times, with relevant evidence attached to the same address.

This anchor provides a shared index: images and conversations can point to the same experience, rather than being pieced together at answer time merely because their keywords are similar.

Layer one: clearly remember what happened at the time

Each event unit retains keyframes, visual descriptions, conversation transcripts and structured fields such as actions, objects, topics, scenes and entities. A discussion of a floral-art plan, for example, records who said what while also retaining the flowers and vase visible in the scene.

Layer two: preserve what came before and after

Thirty seconds alone may not be enough to understand a whole event. EM²Mem therefore builds multiscale contexts at durations such as 3 minutes, 10 minutes and 1 hour, linking text and visual summaries from longer periods back to base events to connect local details with the longer narrative.

Layer three: connect experiences and accumulate long-term patterns

An episodic graph connects people, objects, places, topics and temporal relationships within specific events. A semantic graph further organizes habits, preferences, daily patterns and stable relationships. This higher-level information remains linked to the events supporting it, making its basis traceable.

Article illustration

Figure 2 | From left to right: construct event memory units; connect experiences with episodic and semantic graphs; then retrieve, expand and filter evidence based on the question. Zoom in when reading on a phone.

When a question arrives, the system first locates relevant events, adds clues along temporal or entity relationships, then uses a selector to retain necessary evidence and form a compact answering context. For questions requiring visual verification, a few keyframes can also be added.

This gives the model all of the following when answering: what happened, whom it concerns and where the evidence comes from.

03 | A case: who exactly planned to grow flowers?

The paper uses a multiperson discussion to demonstrate the difference between the two ways of organizing memory.

The question is: Who plans to grow flowers?

Flowers, vases, handicrafts and statements from different participants all appear in the discussion. WorldMM retrieved several flower-related segments, but the connection between the person and the plan remained unclear, ultimately producing the incorrect answer Tasha.

In particular, a Tasha-related segment contained "planning," semantically close to "plans" in the question, but did not explicitly identify whose flower-growing plan it was. This local semantic match may have interfered with the model's judgment about who owned the plan, leading it to mistake a person in a relevant segment for the plan's proposer.

EM²Mem, by contrast, retained the connections among Katrina, the relevant plan, objects and temporal context during memory construction. Reading further into the relevant events let it combine her discussion of the growing cycle with the later statement "we can grow some too," distinguish these from other people's related remarks and arrive at the correct answer: Katrina.

Article illustration

Figure 3 | Both methods find flower-related clues; the key difference is whether the link between a person and the plan is retained. This figure presents a qualitative case from the paper.

This case makes the paper's motivation concrete: with multiple people, objects and time segments, relevant retrieval alone is not enough. Memory organization must also retain relationships within events.

04 | Experiments: more accurate answers and better evidence localization

The study evaluates EM²Mem on three long-video multiple-choice question-answering benchmarks, covering first-person everyday experiences, ultralong first-person reasoning and open-domain long-video understanding.

Article illustration

† marks WorldMM results reproduced in the paper under the same evaluation settings as EM²Mem. The Ego-R1 Bench comparison uses results reported in the original WorldMM paper. WorldMM's original mean accuracies on EgoLifeQA and Video-MME (L) were 65.6% and 76.6%, respectively.

Beyond the final answer, the paper also checks whether the system finds the corresponding evidence segments.

In the strict 30-second event-level evidence-recall evaluation, only a hit on the annotated event segment counts as correct. EM²Mem's Top-5 recall reaches 30.8%, exceeding WorldMM's 23.8% after five iterative retrieval rounds by 7.0 percentage points; its Top-1 recall also reaches 23.0%.

Article illustration

Figure 4 | Event-level evidence recall. The horizontal axis means the number of Top-k events returned for EM²Mem, but up to k retrieval rounds for WorldMM. These are different budget definitions.

This result indicates that event memories are easier to retrieve accurately, and makes memory quality inspectable beyond answer accuracy.

05 | Why structured information and early alignment work

Where do the gains come from? The paper further compares two things: the form in which visual information enters memory, and when multimodal information is aligned.

In an analysis experiment on the first 250 EgoLifeQA questions, the study compared raw frames, flat descriptions and structured event fields. The results show that the combination of structured event fields + alignment during construction achieved the highest accuracy of 71.2%; using the same structured fields but fusing evidence only during retrieval achieved 68.0% accuracy.

Article illustration

Figure 5 | Combining structured event fields with construction-time alignment produced the best result in this set of analysis experiments. This evaluation uses the first 250-question subset and differs from the full-benchmark results above.

Structured fields explicitly retain clues such as people, actions and objects. Early alignment gives these clues a shared event association before retrieval. Together, they make multimodal observations easier to turn into retrievable evidence usable for answering.

06 | Move organization earlier to make repeated question answering lighter

In the EgoLifeQA inference-efficiency comparison, EM²Mem versus WorldMM:

  • Average time per question-answering query fell from 459.00 seconds to 98.21 seconds, a speedup of about 4.67 times.
  • Total inference-stage token use in the evaluation fell from 42.03M to 15.27M, a reduction of 63.66%.

These gains come from already organized event memory: at question time, relevant evidence can be read and filtered, reducing the work of organizing multimodal clues on the fly.

More importantly, memory is built once and can be reused for subsequent queries. As the number of questions grows, construction overhead is gradually amortized, and lower per-query costs translate into end-to-end advantages. In the paper's EgoLifeQA evaluation, including memory construction and 500 question-answering queries, total end-to-end time fell from WorldMM's 318,210 seconds to 105,160 seconds, a speedup of about 3.03 times.

The appendix further analyzes the relationship between query count and cumulative cost. Estimated using the paper's actual evaluation execution procedure, about 24 queries amortize the additional construction time. Counting input and output tokens with equal weight, after about 536 queries, total token use including construction also begins to fall below WorldMM's. Continued reuse of the same memory further shows EM²Mem's advantages in average end-to-end time and token use per query.

Article illustration

Figure 6 | Cumulative overhead including memory construction and subsequent question answering. Bars show total time, lines show total token use, and vertical dashed lines mark the respective crossover points for the two costs. Beyond the corresponding crossover, EM²Mem's cumulative overhead and amortized overhead per query are both below WorldMM's. Time follows the paper's actual evaluation execution procedure; tokens are counted equally by number.

This also fits how long-term memory is used. Personal experiences, video archives and enterprise knowledge are usually queried repeatedly, so event memories organized in advance can provide value through continued use. For questions requiring verification of visual details such as colors or small objects, the system can also return to keyframes to check.

From memory research to applications:

LightMem-Ego, a multimodal agent memory system

Building on EM²Mem, the team further developed LightMem-Ego, a multimodal agent memory system, and provided two application cases bringing multimodal long-term memory into everyday life and research.

Personal-life assistant: help AI remember the little details of life. By recording everyday experiences through phones and smart glasses, AI can help users find casually placed objects, revisit past conversations and organize life notes, creating a personalized assistant.

Article illustration

Scientific-laboratory assistant: help AI remember experiments. By continuously recording experimental video, operations and conversations, AI can help researchers retrace experimental steps, find key operations and experimental phenomena, and organize experiment records to support scientific work.

Article illustration

  • LightMem-Ego technical report: https://arxiv.org/abs/2607.11487
  • See system technical details: https://github.com/zjunlp/LightMem-Ego

LightMem-Ego was jointly developed by Yijun Chen, Boyi Xiao, Yixian Zhao, Yi He, Haoting Xia, Xiangyu Guo, Buqiang Xu and others.

This article comes from "Synced." Author: "Synced."

Related topics

After reading

Join the AINRK community

Read how to join the community and connect with people working in AI.

Community information (opens in a new tab)