mlsanityData to 7 Oct 2026

Domain Mixture Design via Log-Likelihood Differences for Aligning Language Models with a Target Model

arXiv 2603.16622 ↗First posted 17 Mar 2026cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Instead of directly distilling a language model, this study addresses the problem of aligning a base model with a target model in distribution by designing the domain mixture of training data for pretraining or continued pretraining as a fixed training recipe. We propose a method for determining domain weights by viewing models as points in log-likelihood space and aligning the training update direction with the direction toward the target model. Experiments with NanoGPT show that the proposed method consistently reduces the KL divergence between the trained base model and the target model relative to training with Pile-original weighting. Although knowledge distillation remains more effective when available, the proposed method achieves meaningful alignment, and downstream task performance also tends to become closer to that of the target model.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 38 references that are in the map. First posted on arXiv 17 Mar 2026.

Builds on

26 of 38 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality17 Jun
  2. 2023LIMA: Less Is More for Alignment18 May

Other references · 24

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  8. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  9. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  10. 2016Sequence-Level Knowledge Distillationcited by 761
  11. 2025LIMO: Less is More for Reasoningcited by 488
  12. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
12 more
  1. 2024A Survey on Data Selection for Language Modelscited by 230
  2. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 172
  3. 2024CodeGemma: Open Code Models Based on Gemmacited by 145
  4. 2025Organize the Web: Constructing Domains Enhances Pre-Training Data Curationcited by 73
  5. 2025Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-trainingcited by 62
  6. 2024DDK: Distilling Domain Knowledge for Efficient Large Language Modelscited by 25
  7. 2025Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluationcited by 22
  8. 2026Olmix: A Framework for Data Mixing Throughout LM Developmentcited by 12
  9. 2025Why Less is More (Sometimes): A Theory of Data Curationcited by 8
  10. 2025Mapping 1,000+ Language Models via the Log-Likelihood Vectorcited by 7
  11. 2025Establishing a Scale for Kullback-Leibler Divergence in Language Models Across Various Settingscited by 3
  12. 2025Likelihood Variance as Text Importance for Resampling Texts to Map Language Modelscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Mix-CPT: A Domain Adaptation Framework via Decoupling Knowledge Learning and Format Alignment
  2. 2025Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives
  3. 2024A Teacher Is Worth A Million Instructions
  4. 2026A Better Start for Language Models: Domain-Conditional Position Offsets
  5. 2024LIONs: An Empirically Optimized Approach to Align Language Models
  6. 2025IDEAL: Data Equilibrium Adaptation for Multi-Capability Language Model Alignment
  7. 2024Mixture-of-Instructions: Aligning Large Language Models via Mixture Prompting
  8. 2024Hybrid Alignment Training for Large Language Models