mlsanityData to 14 Oct 2026

Reinforcement learning fine-tuning of language model for instruction following and math reasoning

arXiv 2506.21560 ↗First posted 11 Jun 2025cs.CL cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

This study investigates the effectiveness of reinforcement learning (RL) fine-tuning techniques on a compact language model (Qwen2.5-0.5B Base) for two challenging tasks: instruction following and mathematical reasoning. We compare supervised fine-tuning (SFT), Direct Preference Optimization (DPO) using preference-labeled data, and Reinforce Leave-One-Out (RLOO) with reward models. Our experiments show that RLOO with DeBERTa reward modeling achieves the best alignment, while DPO provides strong and consistent results. For math reasoing tasks, synthetic data augmentation and best-of-N sampling with an external verifier significantly improve accuracy, showing the potential of combining fine-tuning with inference-time tools. This study highlights key trade-offs and practical strategies for training lightweight, task-aligned small-scale language models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 14 of its 17 references that are in the map. First posted on arXiv 11 Jun 2025.

Builds on

14 of 17 references.

References · 14

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  4. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  5. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  6. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  7. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,407
  8. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  9. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  10. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 151
  11. 2024Stream of Search (SoS): Learning to Search in Languagecited by 145
  12. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMscited by 108
2 more
  1. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 89
  2. 2023ALJP: An Arabic Legal Judgment Prediction in Personal Status Cases Using Machine Learning Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
  2. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
  3. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
  4. 2026RPO:Reinforcement Fine-Tuning with Partial Reasoning Optimization
  5. 2024Reinforcement Learning for LLM Post-Training: A Survey
  6. 2025Enhancing LLM Reasoning with Iterative DPO: A Comprehensive Empirical Investigation
  7. 2025Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections
  8. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning