# SED-SFT: Selectively Encouraging Diversity in Supervised Fine-Tuning

arXiv: https://arxiv.org/abs/2602.07464 · HTML: https://mlsanity.com/paper/2602.07464/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 18 references that are in the map.
- First posted on arXiv 7 Feb 2026.

## Builds on (most cited first, 9 of 9)

- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems](https://mlsanity.com/paper/2402.14008/) (2024)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Evaluating the Performance of Large Language Models on GAOKAO Benchmark](https://mlsanity.com/paper/2305.12474/) (2023)
- [On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification](https://mlsanity.com/paper/2508.05629/) (2025)
- [On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting](https://mlsanity.com/paper/2508.11408/) (2025)
- [Enhancing Large Language Model Reasoning via Selective Critical Token Fine-Tuning](https://mlsanity.com/paper/2510.10974/) (2025)
- [One-Token Rollout: Guiding Supervised Fine-Tuning of LLMs with Policy Gradient](https://mlsanity.com/paper/2509.26313/) (2025)
- [Debunk the Myth of SFT Generalization](https://mlsanity.com/paper/2510.00237/) (2025)

## Built on by (most cited first, 7 of 7)

- [TailSFT: Filtered Fine-Tuning Improves Post-Training Performance](https://mlsanity.com/paper/2608.25756/) (2026)
- [Selective Off-Policy Reference Tuning with Plan Guidance](https://mlsanity.com/paper/2605.11505/) (2026)
- [OPD Before RL: Warm-Starting Rubric-Based RL with On-Policy Distillation](https://mlsanity.com/paper/2610.02781/) (2026)
- [When Reasoning Narrows the Move: Diversity Collapse in LLM Game Play](https://mlsanity.com/paper/2607.19523/) (2026)
- [Fine-Tuning Fixes Mode Collapse and Over-Dispersion in LLMs](https://mlsanity.com/paper/2609.16454/) (2026)
- [Don't Repeat Yourself: Self-Supervised Fine-Tuning for Coverage](https://mlsanity.com/paper/2609.31688/) (2026)
- [Facts Don't Speak Louder than Words: The Behavioral Essence of Long-CoT Distillation](https://mlsanity.com/paper/2610.17607/) (2026)

## Abstract (arXiv)

Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) has emerged as the standard post-training paradigm for large language models (LLMs). However, the conventional SFT process, driven by Cross-Entropy (CE) loss, often induces mode collapse, where models over-concentrate on specific response patterns. This lack of distributional diversity severely restricts the exploration efficiency required for subsequent RL. While recent studies have attempted to improve SFT by replacing the CE loss, aiming to preserve diversity or refine the update policy, they fail to adequately balance diversity and accuracy, thereby yielding suboptimal performance after RL. To address the mode collapse problem, we propose SED-SFT, which adaptively encourages diversity based on the token exploration space. This framework introduces a selective entropy regularization term with a selective masking mechanism into the optimization objective. Extensive experiments across eight mathematical benchmarks demonstrate that SED-SFT significantly enhances generation diversity with a negligible computational overhead increase compared with CE loss, yielding average improvements of 2.06 and 1.20 points in subsequent RL performance over standard CE-based baselines on Llama-3.2-3B-Instruct and Qwen2.5-Math-7B-Instruct, respectively. The code is publicly available at https://github.com/pppa2019/SED-SFT