JANUS explores fine-tune-then-rectify: Jacobian projection to mitigate catastrophic forgetting

Tsinghua and DiDi Autonomous Driving researchers propose JANUS, using Jacobian null-space projection and multi-step adaptive rectification after fine-tuning. Retention metrics improve on tested 7B/8B models; the necessary and sufficient condition concerns local first-order loss changes.
Editor's verification (November 22, 2026): The JANUS paper (opens in a new tab)'s necessary and sufficient condition for no forgetting concerns a local first-order approximation to the losses on selected historical samples. It does not guarantee that all previous knowledge remains strictly unchanged under arbitrary finite updates. AP1 is a composite retention metric normalized relative to the pretrained model; AP2 is normalized relative to full-parameter fine-tuning. Values above 100% do not mean raw accuracy above 100%, and changes in metric percentage points are not changes in every new task's raw accuracy. Approximately 30 minutes on one A100 is the post-processing time for the paper's specified 7B model, sampling/compression and iteration settings. It excludes the original fine-tuning cost and is not a universal fixed duration. Multi-task continual learning and broader architectures still require validation.
Janus, the god of time in Roman mythology, has two faces and can look into the past and future at once.
Large models, however, often learn new skills only to forget old knowledge.
This phenomenon, known as “catastrophic forgetting,” is a classic challenge in large-model fine-tuning.
Researchers have long tried to impose constraints during training so that models learn new knowledge while forgetting as little of the past as possible.
Now, however, a joint study from Tsinghua University and DiDi Autonomous Driving proposes a different approach:
Rather than carefully preventing forgetting throughout training, let the model finish fine-tuning first and then make targeted corrections to its parameters.
The study's JANUS framework does not change the original fine-tuning process. By post-processing an already fine-tuned model, it can restore some impaired historical knowledge while preserving new-task capabilities as much as possible.
In mathematics and code-task experiments on LLaMA-2-7B, JANUS raised the historical-knowledge retention metrics from 67.55% and 79.71% to 106.16% and 109.83%, respectively, while the new-task capability metric fell by only around one percentage point.
The work has now been accepted by NeurIPS 2026, and the paper and code are publicly available.

- Paper title: Past, Future, All at Once: Mitigating Stability-Plasticity Dilemma via Post-hoc JANUS Rectification
- Paper: https://arxiv.org/abs/2609.19985
- Code: https://github.com/zhengzl18/JANUS
01 Why do large models keep “learning the new and forgetting the old”?
For large models, learning new tasks and retaining old knowledge are often two goals that are difficult to balance.
The former requires plasticity, while the latter requires the model to maintain stability.
The problem is that the two often constrain each other. On one hand, the model must continually adjust its parameters to adapt to new tasks and data. On the other, those parameter changes can interfere with knowledge representations established during pretraining.
Some existing fine-tuning methods for preventing forgetting impose “subspace orthogonality” constraints, restricting parameter updates from directions that might interfere with old knowledge.
The JANUS researchers noticed, however, that these restrictions may be stricter than the conditions actually needed to prevent forgetting.
This is where JANUS found its opening.
02 Unlocking model potential: From a “sufficient condition” to a “necessary and sufficient condition”
The first thing JANUS does is reexamine the mathematical conditions required to prevent forgetting.
The subspace-orthogonality constraint requires the subspaces spanned respectively by the parameter update ΔW and historical knowledge M to be orthogonal, that is: ΔW · M = 0
This does reduce interference, but it also “locks” many directions. As the number of historical samples grows, the space available for learning new tasks shrinks rapidly.
Put simply, this condition is sufficient but not necessary.
JANUS starts from the idea that a network's parameters act together as a whole. Through cooperation, they can share the responsibility for preventing forgetting.
The method therefore changes “each parameter row independently prevents forgetting” into “all parameters cooperate to prevent forgetting,” namely: Tr(ΔW · M) = 0
This seemingly small change implies a crucial shift in perspective:
ΔW · M = 0 concerns the row space of ΔW and the column space of M, requiring orthogonality between these two relatively low-dimensional subspaces. Tr(ΔW · M) = 0 instead treats them as two vectors in a high-dimensional parameter space and constrains the orthogonality between those vectors.
This condition is not only sufficient but also necessary.
The model thus gains more room for parameter updates to learn new tasks while retaining old knowledge.

Subspace orthogonality versus parameter-space orthogonality
On this basis, the researchers designed the JANUS (JAcobian NUll Space Projection) algorithm. Through Jacobian null-space projection, it rectifies fine-tuned parameter updates toward directions that leave old-task losses unchanged to first order.
Notably, this process does not have to be embedded in the original training procedure.
A model can first learn a new task through full-parameter fine-tuning, LoRA or other methods, and then use JANUS to rectify the completed parameter update.
This also distinguishes JANUS from existing training-time methods for preventing forgetting: it attempts to repair fine-tuning's effect on old knowledge after the model has already learned the new task.
03 One correction is not enough: Gradually restoring memory along a “safe path”
Post-training parameter rectification, however, is not as simple as performing a single projection.
The no-forgetting update directions indicated by old-task gradients are only local. This means that when parameter changes are large, the direction initially calculated may no longer be accurate.
A single large projection can still leave the final parameters outside the region where historical losses remain unchanged.
To address this, the researchers further proposed a “multi-step adaptive rectification” mechanism.
Its central idea is to make corrections in “many small steps,” rather than completing the entire correction at once.
After each step, JANUS recalculates the Jacobian at the current position to provide a no-forgetting update direction for the next step.
The algorithm also introduces a JANUS shift metric, comparing changes in the Jacobian's row space at different positions to adaptively choose a suitable step size.
One can think of this as moving along a contour of historical-task loss: take a small step, reconfirm the direction of the contour, then continue with the next step.
The study also introduces techniques such as Ghost operations and SVD compression to reduce the time and space overhead of rectifying large-model parameters.
A model with 7B parameters can be rectified in 30 minutes using just one A100 GPU.

Multi-step adaptive rectification: Taking small steps within a locally trusted region
04 Recovering old knowledge beyond pretrained levels, with a new-capability loss of around one percentage point
How far can JANUS mitigate catastrophic forgetting?
The researchers conducted systematic evaluations on LLaMA-2-7B and LLaMA-3-8B, covering three categories of downstream tasks: mathematics, code and instruction following.
Comparison methods included ordinary fine-tuning methods—full-parameter fine-tuning, LoRA and PiSSA—and methods designed to prevent forgetting: CorDA, MiLoRA and LoRA-Null. JANUS applied to PiSSA's training results is denoted JANUS-P.
The paper uses three composite metrics to measure stability and plasticity: AP₁ measures historical-knowledge retention, or stability; AP₂ measures new-task capability, or plasticity; AP is their arithmetic mean.
JANUS-P achieved particularly striking results on LLaMA-2-7B.

Experimental results on LLaMA-2-7B
On mathematics tasks, JANUS-P raised the stability metric AP₁ from 67.55% to 106.16%, while the plasticity metric AP₂ fell by only 1.02 percentage points. On code tasks, AP₁ rose from 79.71% to 109.83%, while AP₂ fell by only 0.75 percentage points.
On instruction-following tasks, JANUS-P likewise raised stability close to the pretrained model's level and achieved a higher combined metric. Compared with other fine-tuning methods designed to prevent forgetting, JANUS-P achieved improvements in both stability and plasticity in almost all comparisons.

Stability–plasticity comparison
Further experiments showed that this recovery effect occurred consistently across base models, downstream tasks and original fine-tuning methods, indicating that JANUS is not tied to a single training approach.
05 Conclusion: From “training-time constraints” to “post-training repair,” a new approach to continual learning in large models
Perhaps the most noteworthy aspect of JANUS is not only its specific performance gains, but how it changes the way the forgetting problem is addressed.
Previously, many methods sought to avoid damaging old knowledge by restricting parameter updates while the model learned new tasks.
JANUS moves that constraint to after training.
This shift means a mechanism for preventing forgetting need not be deeply tied to a particular fine-tuning algorithm. It could become a relatively independent model post-processing stage.
For large models that must frequently adapt to different tasks and continually update their capabilities, this offers a technical path with practical application potential and demonstrates a new possibility:
The conflict between learning new knowledge and retaining old knowledge in large models need not be resolved only by imposing stricter constraints during training.
Sometimes, letting the model learn first and correcting it afterward may be a more flexible path.
About the author team
The paper's first author, Zhilong Zheng, is a doctoral student in Tsinghua University's Intelligent Driving Research Group (iDLab), supervised by Professor Shengbo Li of Tsinghua's School of Vehicle and Mobility and College of AI. The work was jointly completed by Tsinghua University and DiDi Autonomous Driving's DiDi Voyager Labs.
This article is from the WeChat account “Heart of Machine,” by “Heart of Machine.”
Related topics
After reading
Join the AINRK community
Read how to join the community and connect with people working in AI.




