AI guides

NPU proposes ROSE: Orthogonal editing and knowledge integration reduce interference in multimodal updates

By 机器之心0 viewsSource: 机器之心
ROSE framework for risk-aware orthogonal editing and multimodal knowledge integration

Northwestern Polytechnical University proposes ROSE for lifelong multimodal knowledge editing. Risk-aware orthogonal editing and multi-granularity integration reduce interference; reliability, generalization, and locality are evaluated on specified VLKEB and MMEdit setups.

Factual knowledge in the real world does not remain unchanged: a company's CEO may change, product information is updated as products are released and iterated, and new sporting results emerge continually. Multimodal large models need to absorb this new information promptly to keep their knowledge up to date. However, as new facts are continually introduced, preventing interference between old and new knowledge while retaining as much as possible of the general capabilities developed during pretraining remains a key problem in urgent need of a solution.

To address this challenge, under the joint supervision of Professors Lingyun Song and Xuequn Shang at Northwestern Polytechnical University, the research team proposed the lifelong multimodal knowledge editing framework ROSE. By constraining the direction of parameter updates during knowledge editing, the framework reduces interference between old and new knowledge. During inference, it activates relevant edited knowledge as needed, thereby reducing knowledge forgetting and the degradation of existing capabilities caused by continual knowledge updates. The paper was selected for NeurIPS 2026 Spotlight.

Article illustration

  • Paper title: ROSE: Risk-Aware Orthogonal Subspace Navigation for Lifelong Knowledge Editing in Multimodal Large Language Models
  • Authors: Lingyun Song, Ziyao Chen, Kang Pan, Xiaolin Han, Yifei Zhang, Xiaoqi Wang, Yudai Pan, Xuequn Shang
  • Paper: https://arxiv.org/abs/2611.09770
  • Code: https://github.com/sunlitsong/ROSE

What challenges arise when knowledge is updated continually?

Knowledge editing aims to correct errors or update specific facts without retraining an entire model. In real applications, new facts emerge continually as events unfold, and existing facts may also change. Lifelong multimodal knowledge editing therefore requires a model to learn new knowledge while retaining the results of previous edits and maintaining its original performance on unrelated inputs. This paper groups the challenge into two aspects:

1. Catastrophic Forgetting: Subsequent knowledge updates interfere with the results of earlier edits, making it difficult for the model to retain factual knowledge that has already been edited. For example, the model may have previously been corrected about Apple's CEO, only to answer incorrectly again after a later update.

2. Locality Erosion: Knowledge updates cause unintended changes in a model's outputs about unrelated knowledge, and may even damage its original pretrained capabilities. For example, after information about Apple's CEO is updated, the model may also make unnecessary changes to its answers about the CEOs of other companies.

The connections between visual and textual representations make it even harder to control the scope of an edit. A single factual update may affect both image understanding and language responses. It is therefore necessary to retain historical edits while also maintaining the stability of unrelated knowledge and existing capabilities.

Why consecutive edits are prone to interfering with one another

Knowledge editing seeks to correct specific facts in a model at relatively low cost, avoiding the need to retrain the entire model for every update. For example, a model might receive several knowledge updates in succession: first, changing Apple's CEO from Tim Cook to John Ternus; then updating Ternus's employment history and the business areas he previously oversaw; and finally adding changes in Cook's role after he stepped down as CEO. After these edits, the model must not only have an accurate grasp of each new piece of information, but also ensure that knowledge already written into it is not incorrectly altered by subsequent updates to other facts.

The difficulty, however, is that the knowledge spaces on which different facts depend within a large model are not mutually independent. A new edit may change feature directions on which previous edits rely, and may also affect outputs for unrelated inputs. Thus, even if the model successfully incorporates a new fact, subsequent edits may still cause historical knowledge to be forgotten or change its answers to unrelated questions.

The figure below uses optimization paths to illustrate the problem intuitively. If an update is too conservative, it may fail to edit a new fact successfully. If it is too aggressive, it can easily interfere with other knowledge. ROSE therefore seeks to avoid high-risk parameter subspaces that could damage pretrained capabilities during knowledge editing, while reducing conflicts with the directions of historical edits, striking a balance between writing new knowledge and retaining existing knowledge.

Article illustration

Figure 1. Schematic of optimization paths for traditional knowledge editing methods and ROSE.

How ROSE writes and uses new knowledge

During editing, ROSE freezes the original model's backbone and introduces an auxiliary branch into the FFN down-projection matrix of selected Transformer layers. Continually arriving new knowledge is written into this branch. During inference, the model first determines whether the input concerns knowledge that has already been edited, then decides how to combine the representations of the two branches.

Article illustration

Figure 2. Overview of the ROSE framework.

A risk-aware orthogonal editing mechanism

To reduce interference from subsequent edits with existing modifications, ROSE separately extracts visual and textual features from historical edits, building corresponding knowledge subspaces to record the feature directions involved in those edits. During a new round of editing, ROSE projects the gradient onto the orthogonal complement of the historical knowledge subspace, removing gradient components that overlap with historical editing directions. This allows new parameter updates to avoid, as far as possible, feature directions involved in existing edits, thereby reducing interference between old and new knowledge.

However, protecting only the feature subspaces involved in historical edits is not enough to guarantee the stability of the model's original knowledge and general capabilities. ROSE therefore introduces a dynamic risk-mask mechanism. Using the fixed original weights as an anchor, it selects parameter regions that have changed more substantially to participate in subsequent updates, while suppressing updates at other parameter positions. This mask is dynamically recomputed during gradient updates to further protect the general knowledge and capabilities developed by the large model during pretraining.

A multi-granularity knowledge integration strategy

The edited knowledge in the auxiliary branch is not applicable to every input. Updating information about Apple's CEO, for example, should not affect the model's responses about other, unrelated companies. For each edit, ROSE therefore separately extracts visual and textual semantic prototypes to record the knowledge features involved in that modification. When the model receives a new question, it calculates the similarity between the input features and historical semantic prototypes to determine whether the current input is relevant to an existing edit. Inputs that do not pass the semantic gate directly use the original backbone's output.

For inputs that pass the gate, ROSE further compares the differences between the feature representations of the original backbone and the auxiliary branch, and uses them to construct a dynamic fusion mask. It introduces edited feature representations only at positions where the differences are relatively significant, retaining the original representations elsewhere. This coarse-to-fine approach to knowledge integration enables selective use of edited knowledge, reducing interference with unrelated knowledge and existing capabilities.

ROSE's performance on long editing sequences

The paper tests ROSE on two benchmarks, VLKEB and MMEdit, with tasks including visual question answering and image captioning. Reliability concerns whether the model correctly completes the targeted knowledge update; generalization examines whether the model can still answer accurately when the question's wording or the image changes; locality concerns whether knowledge editing affects other, unrelated inputs, with the aim of keeping the model's outputs on those inputs as consistent as possible with its pre-edit outputs.

The table below shows the results for LLaVA-v1.5-7B after 1,000 consecutive edits on VLKEB.

Article illustration

The results make the following points clear:

1. ROSE effectively mitigates catastrophic forgetting.

As the table shows, the reliability of most baseline methods declines substantially as the number of edits increases. After 1,000 edits on VLKEB, FT-L's reliability falls to 68.14%, while ROSE still reaches 95.24%. From 1 edit to 1,000 edits, ROSE drops by only 4.76 percentage points, compared with 26.15 percentage points for FT-L. This is attributed to the risk-guided orthogonal editing mechanism: orthogonal constraints reduce directional conflicts between old and new edits, while the dynamic risk mask restricts the scope of parameter updates, reducing interference with pretrained knowledge and capabilities.

2. Better knowledge generalization.

On VLKEB, ROSE performs well in both textual and multimodal generalization, outperforming the strongest baseline, LiveEdit, on both after 1,000 edits. Its textual generalization reaches 92.74%, versus 83.97% for LiveEdit, while its multimodal generalization reaches 84.53%, versus 82.75% for LiveEdit. This is attributed to the perturbation-guided knowledge subspace fusion mechanism, which compares feature differences between the two branches, selectively fuses the parts with significant changes, and preserves the remaining original representations.

3. Strong locality.

Preserving the integrity of unrelated knowledge is difficult for many editing methods. SERAC's multimodal locality falls to 15.04% on VLKEB. By contrast, ROSE achieves 100% textual locality and 88.39% multimodal locality. This is attributed to its coarse-to-fine knowledge integration strategy. Semantic alignment gating directs unrelated queries to bypass the auxiliary branch, while perturbation-guided fusion makes fine-grained modifications for relevant inputs, reducing unintended interference with other factual knowledge.

In terms of running costs, under the test configuration using a single RTX 4090 and LLaVA-v1.5-7B, the average time per edit in the 1,000-edit setting is approximately 1.31 seconds, and inference takes approximately 1.13 seconds per query, demonstrating a degree of practical value.

Article illustration

Conclusion

ROSE uses a risk-aware orthogonal editing mechanism to reduce the interference that continual knowledge updates cause to existing knowledge. With a multi-granularity knowledge integration strategy, it calls on edited knowledge as needed and reduces the risk of locality erosion, achieving a better balance between learning new knowledge and retaining existing capabilities. The experimental results further validate the framework's effectiveness in lifelong multimodal knowledge editing tasks.

About the first author

Lingyun Song is an associate professor and PhD supervisor in the School of Computer Science at Northwestern Polytechnical University. His research interests include reinforcement fine-tuning of multimodal large models, agent technology, cross-media cognition and reasoning, and computer vision. He has published more than 60 papers in SCI-indexed journals and leading computer science conferences, and has applied for more than 20 national invention patents and multiple software copyrights.

This article is from the WeChat public account "Machine Heart," by "Machine Heart".

Related topics

After reading

Join the AINRK community

Read how to join the community and connect with people working in AI.

Community information (opens in a new tab)