Why do large models develop massive activations? Tsinghua and Peking study finds a backpropagation clue

Researchers connect attention sinks and massive activations through backpropagation, proposing that RMSNorm lets large activations regulate local gradients. V-scale and backward-only experiments support this explanation, primarily in Llama-style dense models.
Editor's verification (November 22, 2026): Version 3 of the paper (opens in a new tab) primarily analyzes Llama-style dense models with pre-norm/RMSNorm. The authors have not established whether the mechanism transfers to MoE or multimodal architectures, nor have they excluded factors such as data distribution, optimizers and numerical precision. The change from 40.3 to 5.6 in the 0.3B model is a result under the listed intervention settings, not a fixed magnitude across all models. The experiments support this gradient-regulation explanation; they do not show the elimination of all attention sinks or establish a single cause across all architectures.
A recurring phenomenon appears in large language models. A few tokens at the start of a sequence often attract a great deal of attention. Even when they carry little actual meaning, the model still looks toward them frequently. This is known as attention sinks.
Another curious phenomenon often occurs at the same position. Some numerical values produced by the model's internal computations become extremely large, forming massive activations. The two phenomena often occur together, but how exactly are they connected?

Previous research has mainly examined the model's forward computation, observing how it processes inputs and forms outputs. In a study accepted by NeurIPS 2026, however, researchers from Tsinghua University and Peking University turned to the model's other half: backpropagation.
They found that attention sinks also concentrate training signals on a few tokens, forming “gradient sinks.” Meanwhile, massive activations can attenuate these locally concentrated gradients through RMSNorm.
In other words, apparently abnormal large activations may serve a gradient-regulation function.

- Paper title: Attention Sinks Induce Gradient Sinks: Massive Activations as Gradient Regulators in Transformers
- Authors: Yihong Chen, Zhouchen Lin, Quanming Yao
- Paper: https://arxiv.org/abs/2603.17771
- Project code: https://github.com/LARS-research/GradientSink
01 Normalized activations:
Why are they trained to become so large?
To understand the researchers' motivation, we first need to examine the normalization layers in modern large Transformer models. In models using pre-norm, a token's hidden state is first normalized by RMSNorm before entering an attention or feed-forward network (MLP) sublayer. This means that if the vector's direction stays unchanged and it is simply scaled up as a whole, the normalized input passed to the sublayer remains almost unchanged.

A question naturally follows: if forward computation begins with normalization, why does training produce such enormous activation norms?
The answer may lie in normalization's backward pass. The scale removed during forward computation still affects how gradients propagate backward.
02 Attention routes values in the forward pass
And gradients in the backward pass
The forward computation of attention is a weighted sum of value vectors at different positions. During backpropagation, the same attention weights also determine how gradients are distributed back to those positions. If many later tokens attend to the same position, their gradients also return along those connections.

In causal attention, a token near the beginning can be accessed by many subsequent positions. Once it becomes an attention sink, it has the structural conditions to receive large amounts of backward training signal. Of course, the precise strength of concentration also depends on the directions and correlations of those gradients.
The researchers measured models trained from scratch and several open-source models. The results showed widespread gradient concentration at attention inputs after RMSNorm. The first token's gradient norm was much larger than other tokens', typically one to two orders of magnitude above the average for other early positions. This is the local gradient concentration observed in the paper, called a gradient sink. An attention sink is also a gradient sink.

Breaking the upstream computation down further into the query, key and value paths made the differences clearer. The key and value paths showed prominent peaks at the first token, with the value path the most pronounced; the query path was relatively flat. Local gradient concentration exhibited a clear path-specific structure.
03 Massive activations
Become a “pressure-release valve” for gradients
How are gradient sinks connected to massive activations? Answering that requires examining how gradient scale changes throughout the attention sublayer. Experiments found that although the first token's gradient is sharply amplified through the attention sublayer, the scale of its residual-stream gradient remains relatively stable. This is because RMSNorm compresses the gradients concentrated by the attention layer.
The researchers theoretically analyzed RMSNorm's gradient-transmission properties. The central result states that, all else being equal, the larger the input activation scale, the more strongly RMSNorm contracts backward gradients. More precisely, the upper bound on the operator norm of RMSNorm's Jacobian is inversely proportional to the input's root-mean-square scale. This makes activation scale a means of regulating local gradients.
The paper therefore proposes the following chain of mechanisms: attention concentrates on a few positions, producing gradient sinks along the value path, while larger residual-stream activations attenuate the branch gradients passed back to the residual stream through RMSNorm. From this perspective, massive activations can be understood as an adaptive regulatory structure that emerges during model training.
04 How can this be tested?
Give gradients another means of regulation
If this explanation holds, it yields a testable prediction: if a model gains another route for attenuating local gradients, its dependence on massive activations should weaken even if attention sinks remain.
To test this, the researchers proposed V-scale: scaling after the value projection but before attention aggregation according to the magnitude of the value vectors, applying stronger attenuation to the backpropagated gradients corresponding to small-norm values.
There are two direct reasons to choose the value path: it carries the most pronounced gradient peaks, and value norms at attention-sink positions are often small. This transformation offers another route for attenuating locally concentrated gradients.
The experimental results matched the prediction. In the 0.3B model, the median gradient-sink ratio fell from approximately 40.3 to 5.6. At the same time, the first token's residual-stream activation norm dropped substantially: massive activations were significantly reduced, while the model retained clear attention-sink behavior. The reduction in massive activations mainly came from a decrease in MLP output norms.

However, full V-scale still changes the values in the forward pass. To rule out the influence of forward scaling, the researchers designed an additional control that changed only the backward rule.
The backward-only intervention model exhibited the same separation of phenomena: the first token's residual-stream and MLP output norms decreased, while attention sinks remained pronounced. The experiments therefore support the proposition that “massive activations participate in regulating local gradient pressure.”
05 Understanding model anomalies
Also requires examining how training signals flow
The value of this work lies in connecting two frequently discussed phenomena—attention sinks and massive activations—and identifying their link during backpropagation: gradient sinks.
Attention weights determine both forward information aggregation and backward gradient routing. Normalization layers both change the scale of forward representations and influence backward signal transmission. Considering the two together provides a fuller understanding of why certain extreme activations emerge during training.
The study thus offers a new way of thinking: when we observe certain “anomalies” in a model, we might go further and ask what role they play during its training.
About the authors
- Yihong Chen is a postdoctoral researcher in Tsinghua University's Department of Electronic Engineering. With a background in mathematics and scientific computing, his current research focuses on large language models' internal mechanisms, training stability and agent technologies. He aims to understand model behavior through mathematical analysis and experiments and apply mechanistic insights to model training and design.
- Zhouchen Lin is a professor at Peking University's School of Intelligence Science and Technology, a recipient of the National Science Fund for Distinguished Young Scholars, an IEEE Fellow, an IAPR Fellow and a co-corresponding author of this paper. His main research areas include machine learning, computer vision and numerical optimization. He has long studied the theory and methods of learning models, optimization algorithms and deep-network design.
- Quanming Yao is an associate professor and doctoral supervisor in Tsinghua University's Department of Electronic Engineering, an IET Fellow, a BCS Fellow and a co-corresponding author of this paper. His main research concerns scalable, data-efficient machine learning methods, focusing on large-model training dynamics and stability, few-shot and in-context learning, and data-efficient agent learning.
This article is from the WeChat account “Heart of Machine,” by “Heart of Machine.”
Related topics
After reading
Join the AINRK community
Read how to join the community and connect with people working in AI.




