Data to 7 Oct 2026

Learning Dynamics of LLM Finetuning

arXiv 2407.10490 ↗First posted 15 Jul 2024cs.LG cs.AI cs.CLcited by 93

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Learning dynamics, which describes how the learning of specific training examples influences the model's predictions on other examples, gives us a powerful tool for understanding the behavior of deep learning systems. We study the learning dynamics of large language models during different types of finetuning, by analyzing the step-wise decomposition of how influence accumulates among different potential responses. Our framework allows a uniform interpretation of many interesting observations about the training of popular algorithms for both instruction tuning and preference tuning. In particular, we propose a hypothetical explanation of why specific types of hallucination are strengthened after finetuning, e.g., the model might use phrases or facts in the response for question B to answer question A, or the model might keep repeating similar simple phrases when generating responses. We also extend our framework and highlight a unique "squeezing effect" to explain a previously observed phenomenon in off-policy direct preference optimization (DPO), where running DPO for too long makes even the desired outputs less likely. This framework also provides insights into where the benefits of on-policy DPO and other variants come from. The analysis not only provides a novel perspective of understanding LLM's finetuning but also inspires a simple, effective method to improve alignment performance.

Key facts. Cited by 93 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 50 of its 69 references that are in the map. First posted on arXiv 15 Jul 2024.

Builds on

50 of 69 references.

Closest ancestors · 7

Same direction, newest first

Other references · 43

2017Attention Is All You Needcited by 59,932
2017Proximal Policy Optimization Algorithmscited by 14,441
2015Distilling the Knowledge in a Neural Networkcited by 10,076
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
2017Deep reinforcement learning from human preferencescited by 3,846
2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
2023Qwen Technical Reportcited by 3,273
2019The Curious Case of Neural Text Degenerationcited by 2,881
2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582

31 more

2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
2024Self-Rewarding Language Modelscited by 654
2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
2020Early-Learning Regularization Prevents Memorization of Noisy Labelscited by 414
2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 227
2020Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernelcited by 185
2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
2018Language GANs Falling Shortcited by 97
2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuningcited by 90
2019On the Weaknesses of Reinforcement Learning for Neural Machine Translationcited by 82
2020Compositional Languages Emerge in a Neural Iterated Learning Modelcited by 75
2019Stiffness: A New Perspective on Generalization in Neural Networkscited by 46
2020Introduction to Core-sets: an Updated Surveycited by 45
2020Toward Better Generalization Bounds with Locally Elastic Stabilitycited by 41
2019The Local Elasticity of Neural Networkscited by 37
2024Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Spacecited by 33
2022Making Look-Ahead Active Learning Strategies Feasible with Neural Tangent Kernelscited by 25
2022Better Supervisory Signals by Observing Learning Pathscited by 21
2023How to prepare your task head for finetuningcited by 18
2024Bias Amplification in Language Model Evolution: An Iterated Learning Perspectivecited by 17
2023Improving Compositional Generalization Using Iterated Learning and Simplicial Embeddingscited by 15
2024lpNTK: Better Generalisation with Less Data via Sample Interaction During Learningcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 48; 2026: 42.

93 papers.

Sort:

By area:

2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 66
2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 56
2025Do Not Let Low-Probability Tokens Over-Dominate in RL for LLMscited by 43
2025Tracing the Representation Geometry of Language Models from Pretraining to Post-trainingcited by 33
2026Rethinking the Trust Region in LLM Reinforcement Learningcited by 32
2025On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimizationcited by 31
2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
2025Phi-Ground Tech Report: Advancing Perception in GUI Groundingcited by 24
2025ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via α-β-Divergencecited by 21
2026There Will Be a Scientific Theory of Deep Learningcited by 21
2025Implicit Reward as the Bridge: A Unified View of SFT and DPO Connectionscited by 17
2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Modelscited by 17

38 more

2025LLM Unlearning with LLM Beliefscited by 17
2025Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimizationcited by 15
2025On Group Relative Policy Optimization Collapse in Agent Search: The Lazy Likelihood-Displacementcited by 15
2025Why Reinforcement Fine-Tuning Enables MLLMs Preserve Prior Knowledge Better: A Data Perspectivecited by 14
2025MaPPO: Maximum a Posteriori Preference Optimization with Prior Knowledgecited by 14
2024A Survey on Human-Centric LLMscited by 13
2025The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic Viewcited by 13
2026On the Direction of RLVR Updates for LLM Reasoning: Identification and Exploitationcited by 12
2025ComPO: Preference Alignment via Comparison Oraclescited by 11
2025CPGD: Toward Stable Rule-based Reinforcement Learning for Language Modelscited by 11
2025Learning Facts at Scale with Active Readingcited by 11
2025Token Hidden Reward: Steering Exploration-Exploitation in Group Relative Deep Reinforcement Learningcited by 9
2025Value Drifts: Tracing Value Alignment During LLM Post-Trainingcited by 9
2025Behavior Injection: Preparing Language Models for Reinforcement Learningcited by 8
2025Reinforcement Learning Fine-Tuning Enhances Activation Intensity and Diversity in the Internal Circuitry of LLMscited by 8
2025One-Token Rollout: Guiding Supervised Fine-Tuning of LLMs with Policy Gradientcited by 8
2026On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Modelscited by 8
2025Bidirectional LMs are Better Knowledge Memorizers? A Benchmark for Real-world Knowledge Injectioncited by 7
2025TROLL: Trust Regions improve Reinforcement Learning for Large Language Modelscited by 7
2026COLD-Steer: Steering Large Language Models via In-Context One-step Learning Dynamicscited by 7
2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
2025Train Once, Answer All: Many Pretraining Experiments for the Cost of Onecited by 6
2025On the Impossibility of Retrain Equivalence in Machine Unlearningcited by 6
2026Antibody: Strengthening Defense Against Harmful Fine-Tuning for Large Language Models via Attenuating Harmful Gradient Influencecited by 6
2025Proximalized Preference Optimization for Diverse Feedback Types: A Decomposed Perspective on DPOcited by 5
2026Anchored Policy Optimization: Mitigating Exploration Collapse Via Support-Constrained Rectificationcited by 5
2026Bootstrapping Post-training Signals for Open-ended Tasks via Rubric-based Self-play on Pre-training Textcited by 5
2025Aligning Compound AI Systems via System-level DPOcited by 4
2025Scalable Valuation of Human Feedback through Provably Robust Model Alignmentcited by 4
2025StyleBench: Evaluating thinking styles in Large Language Modelscited by 4
2025Optimizing LVLMs with On-Policy Data for Effective Hallucination Mitigationcited by 4
2026Confidence Calibration for Multimodal LLMs: An Empirical Study through Medical VQAcited by 4
2025Fine-Tuning is Subgraph Search: A New Lens on Learning Dynamicscited by 3
2025Towards Revealing the Effectiveness of Small-Scale Fine-tuning in R1-style Reinforcement Learningcited by 3
2025Coresets from Trajectories: Selecting Data via Correlation of Loss Differencescited by 3
2025Test-Time Policy Adaptation for Enhanced Multi-Turn Interactions with LLMscited by 3
2025Distributional Biases in Post-Training: A Markovian Analysis of Reasoning Trajectoriescited by 3
2026Discern Truth from Falsehood: Reducing Over-Refusal via Contrastive Refinementcited by 3

50 most cited of 93 All 93 (JSON)

Similar, not cited

Close in wording, no citation either way