mlsanityData to 7 Oct 2026

An Empirical Study of SFT-DPO Interaction and Parameterization in Small Language Models

arXiv 2603.20100 ↗First posted 20 Mar 2026cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Direct Preference Optimization (DPO) is widely used after supervised fine-tuning (SFT) to align language models, yet empirical behavior under small backbones and modest data is under-specified. We systematically compare SFT-only, DPO-only, and staged SFT-to-DPO training alongside full fine-tuning (FFT) versus LoRA on a GPT-2-scale decoder, evaluating paraphrase detection and Shakespearean sonnet continuation. DPO yields small, task-dependent gains over strong SFT and can match competitive SFT accuracy without a warm start when the preference construction closely parallels the supervised objective. In contrast, parameterization dominates: FFT consistently outperforms LoRA at matched training depth, and LoRA does not reduce wall-clock time on our hardware. These findings indicate that, in this small-scale regime, supervised full-parameter adaptation remains the primary performance lever, while preference optimization and low-rank adaptation provide limited marginal returns.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 4 of its 6 references that are in the map. First posted on arXiv 20 Mar 2026.

Builds on

4 of 6 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 3

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2018Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learningcited by 57

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPO
  2. 2024Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process
  3. 2025Balancing the Budget: Understanding Trade-offs Between Supervised and Preference-Based Finetuning
  4. 2024Intermediate direct preference optimization
  5. 2024Preference-Oriented Supervised Fine-Tuning: Favoring Target Model Over Aligned Large Language Models
  6. 2025Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections
  7. 2024Insights into Alignment: Evaluating DPO and its Variants Across Multiple Tasks
  8. 2024ORPO: Monolithic Preference Optimization without Reference Model