mlsanityData to 14 Oct 2026

Diversity in Large Language Models under Supervised Fine-Tuning

arXiv 2605.00195 ↗First posted 30 Apr 2026cs.LGcited by 4

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Supervised Fine-Tuning (SFT) is essential for aligning Large Language Models (LLMs) with user intent, yet it is believed to suppress generative diversity. Although this reduction is frequently referenced, formal empirical testing of the phenomenon remains limited. The expressiveness of LLMs by itself was addressed by multiple prior methods. Their varying perspectives suggest that deeper investigation could yield further improvements. In this study, we attribute the decline to two primary drivers: the neglect of low-frequency patterns within fine-tuning datasets and the forgetting of preexisting knowledge. Motivated by our theoretical analysis, we develop Tempered Focal (TOFU) loss, a novel objective that addresses both stated challenges simultaneously. Our extensive evaluation confirms at scale that generation breadth narrows after SFT and strengthens the hypothesis explaining this effect. Across multiple models and benchmarks, we demonstrate that TOFU enhances output diversity while preserving high response quality, offering a principled approach to SFT.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 42 of its 59 references that are in the map. First posted on arXiv 30 Apr 2026.

Builds on

42 of 59 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  12. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
29 more
  1. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  6. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  7. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  8. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  9. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  10. 2024OLMo: Accelerating the Science of Language Modelscited by 613
  11. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 491
  12. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 451
  13. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  14. 2020Calibrating Deep Neural Networks using Focal Losscited by 357
  15. 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 150
  16. 2017Controlling Linguistic Style Aspects in Neural Language Generationcited by 149
  17. 2022Rethinking with Retrieval: Faithful Large Language Model Inferencecited by 145
  18. 2020Improved Natural Language Generation via Loss Truncationcited by 75
  19. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 70
  20. 2025NoveltyBench: Evaluating Language Models for Humanlike Diversitycited by 66
  21. 2020Text Generation by Learning from Demonstrationscited by 64
  22. 2025Scaling Laws for Forgetting during Finetuning with Pretraining Data Injectioncited by 35
  23. 2023Tailoring Language Generation Models under Total Variation Distancecited by 27
  24. 2020On Focal Loss for Class-Posterior Probability Estimation: A Theoretical Perspectivecited by 21
  25. 2025Control the Temperature: Selective Sampling for Diverse and High-Quality LLM Outputscited by 15
  26. 2025Influences on LLM Calibration: A Study of Response Agreement, Loss Functions, and Prompt Stylescited by 12
  27. 2025Understanding Overadaptation in Supervised Fine-Tuning: The Role of Ensemble Methodscited by 9
  28. 2025Improving Diversity in Language Models: When Temperature Fails, Change the Losscited by 7
  29. 2024Beyond Accuracy Optimization: Computer Vision Losses for Large Language Model Fine-Tuningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Multimodal Alignment and Preference Optimization for Zero-Shot Conditional RNA Generationcited by 0
  2. 2026Fine-Tuning Fixes Mode Collapse and Over-Dispersion in LLMscited by 0
  3. 2026Strategically Diverse Sampling for Self-Trainingcited by 0
  4. 2026Don't Repeat Yourself: Self-Supervised Fine-Tuning for Coveragecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Massive Supervised Fine-tuning Experiments Reveal How Data, Layer, and Training Factors Shape LLM Alignment Quality
  2. 2024Rethinking Data Selection at Scale: Random Selection is Almost All You Need
  3. 2025Efficient Response Generation Strategy Selection for Fine-Tuning Large Language Models Through Self-Aligned Perplexity
  4. 2024Scaling Sparse Fine-Tuning to Large Language Models
  5. 2024Navigating the Landscape of Large Language Models: A Comprehensive Review and Analysis of Paradigms and Fine-Tuning Strategies
  6. 2024Prefix Text as a Yarn: Eliciting Non-English Alignment in Foundation Language Model
  7. 2025Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data
  8. 2025Complexity-aware fine-tuning