mlsanityData to 14 Oct 2026

Offline RL by Reward-Weighted Fine-Tuning for Conversation Optimization

arXiv 2506.06964 ↗First posted 8 Jun 2025cs.CL cs.LGcited by 10

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Offline reinforcement learning (RL) is a variant of RL where the policy is learned from a previously collected dataset of trajectories and rewards. In our work, we propose a practical approach to offline RL with large language models (LLMs). We recast the problem as reward-weighted fine-tuning, which can be solved using similar techniques to supervised fine-tuning (SFT). To showcase the value of our approach, we apply it to learning short-horizon question-answering policies of a fixed length, where the agent reasons about potential answers or asks clarifying questions. Our work stands in a stark contrast to state-of-the-art methods in this domain, based on SFT and direct preference optimization, which have additional hyper-parameters and do not directly optimize for rewards. We compare to them empirically, and report major gains in both optimized rewards and language quality.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 55 of its 78 references that are in the map. First posted on arXiv 8 Jun 2025.

Builds on

55 of 78 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May

Other references · 54

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  9. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  11. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  12. 2023Reasoning with Language Model is Planning with World Modelcited by 861
38 more
  1. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  2. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  3. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 672
  4. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 301
  5. 2024V-STaR: Training Verifiers for Self-Taught Reasonerscited by 217
  6. 2019CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databasescited by 185
  7. 2015Doubly Robust Policy Evaluation and Optimizationcited by 158
  8. 2022Offline RL for Natural Language Generation with Implicit Language Q Learningcited by 138
  9. 2023MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problemscited by 131
  10. 2023We're Afraid Language Models Aren't Modeling Ambiguitycited by 126
  11. 2019A Short Note on Concentration Inequalities for Random Vectors with SubGaussian Normcited by 106
  12. 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 104
  13. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 93
  14. 2022DialFRED: Dialogue-Enabled Agents for Embodied Instruction Followingcited by 63
  15. 2024Ask-before-Plan: Proactive Language Agents for Real-World Planningcited by 60
  16. 2024Learning to Clarify: Multi-turn Conversations with Action-Based Contrastive Self-Trainingcited by 57
  17. 2025The Best Instruction-Tuning Data are Those That Fitcited by 52
  18. 2024Exploring Knowledge Tracing in Tutor-Student Dialogues using LLMscited by 46
  19. 2025Improving LLM Safety Alignment with Dual-Objective Optimizationcited by 43
  20. 2025Training LLM-based Tutors to Improve Student Learning Outcomes in Dialoguescited by 43
  21. 2023Zero-Shot Goal-Directed Dialogue via RL on Imagined Conversationscited by 38
  22. 2024Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertaintycited by 25
  23. 2017End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradientcited by 23
  24. 2022EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RLcited by 23
  25. 2025Active Task Disambiguation with LLMscited by 23
  26. 2024CLARINET: Augmenting Language Models to Ask Clarification Questions for Retrievalcited by 20
  27. 2019Imitation-Regularized Offline Learningcited by 17
  28. 2023Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Modelscited by 17
  29. 2025Upweighting Easy Samples in Fine-Tuning Mitigates Forgettingcited by 16
  30. 2025AskToAct: Enhancing LLMs Tool Use via Self-Correcting Clarificationcited by 13
  31. 2024The Imperative of Conversation Analysis in the Era of LLMs: A Survey of Tasks, Techniques, and Trendscited by 11
  32. 2024Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuningcited by 10
  33. 2025Process-Supervised LLM Recommenders via Flow-guided Tuningcited by 10
  34. 2022Local Policy Improvement for Recommender Systemscited by 9
  35. 2022Simulating and Modeling the Risk of Conversational Searchcited by 6
  36. 2024Towards a Zero-Data, Controllable, Adaptive Dialog Systemcited by 4
  37. 2024RL-STaR: Theoretical Analysis of Reinforcement Learning Frameworks for Self-Taught Reasonercited by 4
  38. 2025ImageChain: Advancing Sequential Image-to-Text Reasoning in Multimodal Large Language Modelscited by 4

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 9.

10 papers.

  1. 2026Good SFT Optimizes for SFT, Better SFT Prepares for Reinforcement Learningcited by 11
  2. 2026Reference-Sampled Boltzmann Projection for KL-Regularized RLVR: Target-Matched Weighted SFT, Finite One-Shot Gaps, and Policy Mirror Descentcited by 1
  3. 2026DRIFT: Decoupled Rollouts and Importance-Weighted Fine-Tuning for Efficient Multi-Turn Optimizationcited by 1
  4. 2025Learning to Reason in LLMs by Expectation Maximizationcited by 0
  5. 2026Partial Policy Gradients for RL in LLMscited by 0
  6. 2026Robust Post-Training for Generative Recommenders: Why Exponential Reward-Weighted SFT Outperforms RLHFcited by 0
  7. 2026CuraLight: Debate-Guided Data Curation for LLM-Centered Traffic Signal Controlcited by 0
  8. 2026Children's English Reading Story Generation via Supervised Fine-Tuning of Compact LLMs with Controllable Difficulty and Safetycited by 0
  9. 2026Goal-Conditioned Supervised Learning for LLM Fine-Tuningcited by 0
  10. 2026TriAlign: Towards Universal Truth Consistency in Personalized LLM Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions
  2. 2025Exploring the Potential of Offline RL for Reasoning in LLMs: A Preliminary Study
  3. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
  4. 2025When Actions Teach You to Think: Reasoning-Action Synergy via Reinforcement Learning in Conversational Agents
  5. 2025Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewards
  6. 2024Balancing Enhancement, Harmlessness, and General Capabilities: Enhancing Conversational LLMs with Direct RLHF
  7. 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
  8. 2025FrugalRAG: Less is More in RL Finetuning for Multi-Hop Question Answering