AI guides

NICT's OptiMer searches continual-pretraining mixtures by merging distribution vectors

By 宋海越0 viewsSource: 机器之心
OptiMer distribution-vector merging and mixture optimization

OptiMer first trains separate domain models, then adjusts vector-merging weights to avoid repeated training. Experiments report faster mixture optimization, while the initial domain-training cost remains.

How much should a model train on each type of data? Every model trainer faces this question.

Getting a training mixture wrong often means starting over. How much of each dataset to include in the training set has to be decided before training begins. Whether those proportions are good, however, can only be known after training has finished and the model has been evaluated. Poor results may require adjusting ratios and retraining, making experimentation expensive.

Haiyue Song and colleagues at Japan's National Institute of Information and Communications Technology (NICT) address this in OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training, proposing distribution-vector merging so mixture experiments need not retrain models from scratch.

The paper received an EMNLP 2026 Outstanding Paper Award, one of 22 recipients among 17,661 submissions.

Article illustration

  • Paper: OptiMer: Optimal Distribution Vector Merging Is Better than Data Mixing for Continual Pre-Training
  • Authors: Haiyue Song, Huy Hien Vu, Masao Utiyama
  • Website: https://shyyhs.github.io/optimer

Article illustration

Method | From merging datasets to merging distribution vectors

We propose optimal merging: OptiMer.

Step one: train separate models on each data category. Starting from the base model, we perform continual pretraining separately on each dataset. For example, we train one model on Japanese data, one on Chinese data and one on mathematics data, obtaining three models.

Step two: extract distribution vectors recording training-induced changes. Subtracting the base model's parameters from the parameters of each trained model gives the corresponding distribution vector. It represents the changes that occurred in the model after training on that dataset.

Step three: automatically search for optimal merging proportions. Adding several distribution vectors in different proportions, along with an instruction-following vector, to the base model's weights produces a new model. Based on its performance on the validation set, we can continually optimize the proportions of the distribution vectors to achieve the best merge. Each attempt involves only model merging and evaluation, with no further training required.

Article illustration

Efficiency | 15–35x faster

After initial training on each dataset, trying mixtures conventionally requires 129.3 hours to train just one ratio. OptiMer tries 100 ratios in 8.5 hours. In our experiments, mixture optimization achieved a 15–35x speedup.

Article illustration

Performance | Better than mixed-data training

Across common benchmarks including C-Eval, Japanese Leaderboard, MMLU, GSM8K and HumanEval, OptiMer averages 70.37 versus 63.71 for conventional mixing, an improvement of about 6.7 points.

We also found that the optimal merging weights identified by OptiMer can guide the data proportions used in a data-mixing scheme. When we used the merging weights directly as the data-mixing proportions and retrained the model, the average score rose from 63.71 with equal-proportion mixing to 68.66.

Article illustration

Flexibility | Rapid reoptimization for new tasks

For a new task, traditional data-mixing methods require adjusting the proportions and training again. This process is extremely costly, preventing traditional methods from adapting quickly to new tasks. Our method, however, can reuse distribution vectors that have already been trained; it only needs a different evaluation objective and an automatic adjustment of the merging weights. For example, optimizing for English tasks gives the highest score on the corresponding tasks; optimizing again for mathematics can produce good performance on mathematics tasks; and optimizing again for Japanese tasks can quickly produce a model specialized for Japanese.

This situation is quite common in industrial deployment. If a company wants to develop specialized models for different domains, such as education or finance, traditional methods require a great deal of trial and error with training-data proportions. Our method, by contrast, can automatically and quickly produce specialized models.

Article illustration

Analysis 1 | Why distribution vectors can be merged

Will capabilities learned from different data interfere with one another during merging? We calculated the cosine similarity between every pair of distribution vectors. We found that the cosine similarity between distribution vectors and the instruction-following vector was only about 0.03, close to perpendicular, while the cosine similarity between different distribution vectors was about 0.3. We hypothesize that existing merging algorithms, such as DARE, can combine different vectors with relatively little loss.

Article illustration

Analysis 2 | Principal component analysis

Using principal component analysis, we projected the high-dimensional distribution vectors onto a two-dimensional plane. Because each is the difference between the parameters of two models, its dimensionality equals the number of model parameters. First, the distribution vectors obtained through continual pretraining on different data lie far from the instruction-following vector. Then, from the bar chart of OptiMer's merging weights, we found that OptiMer tends to assign larger weights to the instruction-following vector and smaller weights to the data distribution vectors. As a result, the optimal model produced by OptiMer's merging actually lies near the line connecting the Base model and the IT model; we can imagine drawing a line from PT to IT. We hypothesize that good models should lie near this line.

Article illustration

Analysis 3 | OptiMer's search

We examine how it finds optimal weights. The left chart shows weight space; the right shows score improvement through search.

The left-hand plot shows that high-scoring combinations are concentrated in a relatively narrow region, where the instruction-following vector has a larger weight and the data distribution vectors have smaller weights. This demonstrates the advantage of OptiMer's Bayesian optimization over grid search: grid search is prone to missing regions, especially when there are more datasets. OptiMer, however, needs only roughly 10*n searches, where n is the number of datasets.

The right-hand plot shows that, based on the data points already available, OptiMer's Bayesian optimization gradually concentrates subsequent searches in more promising regions. The algorithm we use supports parallel search, searching eight points at a time. It focuses on exploring the entire space in the early stage, converges rapidly in the middle stage, and finds the highest score of this search run in the late stage.

Article illustration

Analysis 4 | Vector changes during training

We recorded models at different training steps during continual pretraining to track how they changed. The main plot shows the trajectories of these vectors in the PCA plane, while the smaller plots show how vector length and model score change with the number of training steps.

The left-hand plot shows that the trajectory of continual pretraining is approximately a straight line. Increasing or decreasing the weight of a distribution vector is similar to increasing or decreasing the effective amount of training on that dataset.

The right-hand plot shows that, as training progresses, the norm of the distribution vector keeps growing, while model performance peaks early in training and then gradually declines. We hypothesize that this suggests the model should not move too far away from the base model. The same hypothesis also appeared in Analysis 2.

Article illustration

Analysis 5 | Negative weights

The weights of distribution vectors can be set to negative values. We found that this sometimes even produces better results. Does this mean OptiMer can perform machine unlearning, selectively forgetting certain knowledge in the data and thereby achieving better performance on the target task? This paper does not investigate the question further, but we consider it a very interesting one.

Article illustration

Application | A Japanese-specialized model

We apply OptiMer to AI Singapore's Gemma-SEA-LION-v4-27B, based on Gemma 3 and continually pretrained on Southeast Asian languages.

The goal is better Japanese while retaining Southeast Asian language capabilities.

We continually pretrain on Japanese, Chinese, English, mathematics and code separately, extract distribution vectors, merge them with SEA-LION's instruction-following vector and search weights for Japanese-task performance.

Japanese scores rise from 66.34 to 74.40 while Southeast Asian task performance is preserved.

Article illustration

Conclusion

OptiMer adapts open models to target languages or domains more efficiently, effectively and flexibly than conventional mixing.

At the same time, this work conducted experiments only on continual pretraining. In the future, we hope to extend this approach to post-training to solve the problem of finding optimal proportions for the large number of datasets involved in post-training.

First author

Haiyue Song is a post-training engineer in Preferred Networks' language-model development department in Japan, formerly at NICT in Kyoto. He earned a Kyoto University informatics doctorate in 2024 under Professor Sadao Kurohashi and studied computer science and Japanese at Shanghai Jiao Tong University as an undergraduate. His research covers language models and machine translation.

This work was completed while the first author was at NICT.

This article comes from Synced. Author: Haiyue Song.

After reading

Join the AINRK community

Read how to join the community and connect with people working in AI.

Community information (opens in a new tab)