Data to 7 Oct 2026

Preserving Diversity in Supervised Fine-Tuning of Large Language Models

arXiv 2408.16673 ↗First posted 29 Aug 2024cs.LG cs.AIcited by 68

Not on the map. Why?

Cited by 21 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large Language Models (LLMs) typically rely on Supervised Fine-Tuning (SFT) to specialize in downstream tasks, with the Cross Entropy (CE) loss being the de facto choice. However, CE maximizes the likelihood of observed data without accounting for alternative possibilities. As such, CE usually leads to reduced diversity in the model's outputs, which hinders further development that requires sampling to explore better responses. To address this limitation, this paper introduces a new game-theoretic formulation for SFT. In this framework, an auxiliary variable is introduced to regulate the learning process. We prove that the proposed game-theoretic approach connects to the problem of reverse KL minimization with entropy regularization. This regularization prevents over-memorization of training data and promotes output diversity. To implement this framework, we develop GEM, a new training algorithm that is computationally efficient as CE by leveraging some unique properties of LLMs. Empirical studies of pre-trained models from 3B to 70B parameters show that GEM achieves comparable downstream performance to CE while significantly enhancing output diversity. This increased diversity translates to performance gains in test-time compute scaling for chat and code generation tasks. Moreover, we observe that preserving output diversity has the added benefit of mitigating forgetting, as maintaining diverse outputs encourages models to retain pre-trained knowledge throughout the training process.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 50 of its 68 references that are in the map. First posted on arXiv 29 Aug 2024.

Builds on

50 of 68 references.

References · 50

2017Attention Is All You Needcited by 59,932
2015Deep Residual Learning for Image Recognitioncited by 51,809
2020Language Models are Few-Shot Learnerscited by 32,199
2023GPT-4 Technical Reportcited by 18,132
2022Training language models to follow instructions with human feedbackcited by 15,712
2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
2024The Llama 3 Herd of Modelscited by 14,192
2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
2021Training Verifiers to Solve Math Word Problemscited by 9,743
2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
2021Evaluating Large Language Models Trained on Codecited by 8,053
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916

38 more

2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
2022Scaling Instruction-Finetuned Language Modelscited by 2,997
2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
2016Generative Adversarial Imitation Learningcited by 1,576
2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
2018An Algorithmic Perspective on Imitation Learningcited by 455
2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 384
2023What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuningcited by 354
2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
2018The relativistic discriminator: a key element missing from standard GANcited by 234
2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
2024Nemotron-4 340B Technical Reportcited by 143
2024Why Transformers Need Adam: A Hessian Perspectivecited by 123
2019Imitation Learning as f-Divergence Minimizationcited by 121
2023NEFTune: Noisy Embeddings Improve Instruction Finetuningcited by 99
2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 97
2023Amortizing intractable inference in large language modelscited by 94
2020Error Bounds of Imitating Policies and Environmentscited by 90
2024BOND: Aligning LLMs with Best-of-N Distillationcited by 73
2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
2018Maximum-Entropy Fine-Grained Classificationcited by 64
2024Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignmentcited by 32
2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learningcited by 27
2020Towards a Better Global Loss Landscape of GANscited by 17
2024Q-Adapter: Customizing Pre-trained LLMs to New Preferences with Forgetting Mitigationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 27; 2026: 36.

68 papers.

Sort:

By area:

2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wildcited by 505
2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 79
2025NoveltyBench: Evaluating Language Models for Humanlike Diversitycited by 64
2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
2025Diversity-Aware Policy Optimization for Large Language Model Reasoningcited by 39
2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
2025Rethinking Fine-Tuning when Scaling Test-Time Compute: Limiting Confidence Improves Mathematical Reasoningcited by 28
2025Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approachcited by 25
2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Rewardcited by 25
2024Imitating Language via Scalable Inverse Reinforcement Learningcited by 21

38 more

2025Post-training Large Language Models for Diverse High-Quality Responsescited by 19
2025Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPOcited by 18
2026Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoningcited by 18
2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Modelscited by 17
2025Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuumcited by 16
2025CoRT: Code-integrated Reasoning within Thinkingcited by 15
2025Proximal Supervised Fine-Tuningcited by 15
2025SpeechJudge: Towards Human-Level Judgment for Speech Naturalnesscited by 15
2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
2025Self-Evolving Critique Abilities in Large Language Modelscited by 12
2025Reasoning Path Divergence: A New Metric and Curation Strategy to Unlock LLM Diverse Thinkingcited by 10
2026Where does output diversity collapse in post-training?cited by 10
2024Growing a Tail: Increasing Output Diversity in Large Language Modelscited by 9
2025Quality-Diversity Red-Teaming: Automated Generation of High-Quality and Diverse Attackers for Large Language Modelscited by 8
2025Debunk the Myth of SFT Generalizationcited by 8
2025What Does It Take to Be a Good AI Research Agent? Studying the Role of Ideation Diversitycited by 7
2026Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Modelscited by 7
2025Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theorycited by 6
2024Foundation Models for Low-Resource Language Education (Vision Paper)cited by 5
2025Controlling Large Language Model with Latent Actionscited by 5
2025Building Coding Agents via Entropy-Enhanced Multi-Turn Preference Optimizationcited by 5
2026LLMs Exhibit Significantly Lower Uncertainty in Creative Writing Than Professional Writerscited by 4
2026Diversity in Large Language Models under Supervised Fine-Tuningcited by 4
2026Annotations Mitigate Post-Training Mode Collapsecited by 4
2025LM-mixup: Text Data Augmentation via Language Model based Mixupcited by 3
2026Time-Annealed Perturbation Sampling: Diverse Generation for Diffusion Language Modelscited by 3
2026Letting Tutor Personas Speak Up for LLMs: Learning Steering Vectors from Dialogue via Preference Optimizationcited by 3
2026When RL Fails after SFT: Rejuvenating Model Plasticity for Robust SFT-to-RL Handoffcited by 3
2026Sampling More, Getting Less: Calibration is the Diversity Bottleneck in LLMscited by 2
2026TailSFT: Filtered Fine-Tuning Improves Post-Training Performancecited by 2
2025Differential Information Distribution: A Bayesian Perspective on Direct Preference Optimizationcited by 1
2025Rethinking the Evaluation of Alignment Methods: Insights into Diversity, Generalisation, and Safetycited by 1
2026Not All Layers Need Tuning: Selective Layer Restoration Recovers Diversitycited by 1
2026Quality-constrained Entropy Maximization Policy Optimization for LLM Diversitycited by 1
2026D5P4: Partition Determinantal Point Process for Diversity in Parallel Discrete Diffusion Decodingcited by 1
2026Self-Consolidating Language Models: Continual Knowledge Incorporation from Contextcited by 1
2026PriFT: Prior-Support Guided Supervised Fine-Tuningcited by 1
2026BaRA: Bayesian Adaptive Rank Allocation for Parameter-Efficient Fine-Tuningcited by 1

50 most cited of 68 All 68 (JSON)

Similar, not cited

Close in wording, no citation either way