mlsanityData to 14 Oct 2026

Capturing Nuanced Preferences: Preference-Aligned Distillation for Small Language Models

arXiv 2502.14272 ↗First posted 20 Feb 2025cs.CL cs.AIcited by 8

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 2 more
Abstract (arXiv)

Aligning small language models (SLMs) with human values typically involves distilling preference knowledge from large language models (LLMs). However, existing distillation methods model preference knowledge in teacher LLMs by comparing pairwise responses, overlooking the extent of difference between responses. This limitation hinders student SLMs from capturing the nuanced preferences for multiple responses. In this paper, we propose a Preference-Aligned Distillation (PAD) framework, which models teacher's preference knowledge as a probability distribution over all potential preferences, thereby providing more nuanced supervisory signals. Our insight in developing PAD is rooted in the demonstration that language models can serve as reward functions, reflecting their intrinsic preferences. Based on this, PAD comprises three key steps: (1) sampling diverse responses using high-temperature; (2) computing rewards for both teacher and student to construct their intrinsic preference; and (3) training the student's intrinsic preference distribution to align with the teacher's. Experiments on four mainstream alignment benchmarks demonstrate that PAD consistently and significantly outperforms existing approaches, achieving over 20% improvement on AlpacaEval 2 and Arena-Hard, indicating superior alignment with human preferences. Notably, on MT-Bench, using the textscGemma model family, the student trained by PAD surpasses its teacher, further validating the effectiveness of our PAD.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 37 references that are in the map. First posted on arXiv 20 Feb 2025.

Builds on

24 of 37 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec

Other references · 22

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  6. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  7. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  8. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  9. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  10. 2016Sequence-Level Knowledge Distillationcited by 801
  11. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 752
  12. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
10 more
  1. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 308
  2. 2023Preference Ranking Optimization for Human Alignmentcited by 285
  3. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 203
  4. 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 156
  5. 2024Direct Preference Optimization with an Offsetcited by 122
  6. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  7. 2020MixKD: Towards Efficient Distillation of Large-scale Language Modelscited by 65
  8. 2021Scalable Bayesian Inverse Reinforcement Learningcited by 56
  9. 2023Do Not Blindly Imitate the Teacher: Using Perturbed Loss for Knowledge Distillationcited by 25
  10. 2022Teacher Forcing Recovers Reward Functions for Text Generationcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 5.

8 papers.

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025InfiFPO: Implicit Model Fusion via Preference Optimization in Large Language Modelscited by 8
  3. 2026FeatCal: Feature Calibration for Post-Merging Modelscited by 5
  4. 2025Reference-Free Rating of LLM Responses via Latent Informationcited by 4
  5. 2026Probing to Refine: Reinforcement Distillation of LLMs via Explanatory Inversioncited by 1
  6. 2026TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Modelscited by 1
  7. 2026Smaller Models, Better Rejects: Preference Distillation Scalingcited by 1
  8. 2026SMAT: Simple and Efficient Merge-Aware Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Advantage-Guided Distillation for Preference Alignment in Small Language Models
  2. 2024Towards a Unified View of Preference Learning for Large Language Models: A Survey
  3. 2026RM-Distiller: Exploiting Generative LLM for Reward Model Distillation
  4. 2024A Systematic Examination of Preference Learning through the Lens of Instruction-Following
  5. 2026CTPD: Cross Tokenizer Preference Distillation
  6. 2024Transforming and Combining Rewards for Aligning Large Language Models
  7. 2024Understanding the Learning Dynamics of Alignment with Human Feedback
  8. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMs