mlsanityData to 7 Oct 2026

Expanding the Capabilities of Reinforcement Learning via Text Feedback

arXiv 2602.02482 ↗First posted 2 Feb 2026cs.LGcited by 49

Not on the map. Why?

Abstract (arXiv)

The success of RL for LLM post-training stems from an unreasonably uninformative source: a single bit of information per rollout as binary reward or preference label. At the other extreme, distillation offers dense supervision but requires demonstrations, which are costly and difficult to scale. We study text feedback as an intermediate signal: richer than scalar rewards, yet cheaper than complete demonstrations. Textual feedback is a natural mode of human interaction and is already abundant in many real-world settings, where users, annotators, and automated judges routinely critique LLM outputs. Towards leveraging text feedback at scale, we formalize a multi-turn RL setup, RL from Text Feedback (RLTF), where text feedback is available during training but not at inference. Therefore, models must learn to internalize the feedback in order to improve their test-time single-turn performance. To do this, we propose two methods: Self Distillation (RLTF-SD), which trains the single-turn policy to match its own feedback-conditioned second-turn generations; and Feedback Modeling (RLTF-FM), which predicts the feedback as an auxiliary objective. We provide theoretical analysis on both methods, and empirically evaluate on reasoning puzzles, competition math, and creative writing tasks. Our results show that both methods consistently outperform strong baselines across benchmarks, highlighting the potential of RL with an additional source of rich supervision at scale.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Builds on 93 of its 114 references that are in the map. First posted on arXiv 2 Feb 2026.

Builds on

93 of 114 references.

References · 93

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2025Qwen3 Technical Reportcited by 8,195
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2023Let's Verify Step by Stepcited by 3,598
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  12. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
38 more
  1. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  2. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  4. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  5. 2023Mastering Diverse Domains through World Modelscited by 1,193
  6. 2017Hindsight Experience Replaycited by 1,096
  7. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  8. 2020Mastering Atari with Discrete World Modelscited by 990
  9. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  10. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  11. 2016Sequence-Level Knowledge Distillationcited by 761
  12. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  13. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  14. 2022Large Language Models Can Self-Improvecited by 656
  15. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  16. 2013Do Deep Nets Really Need to be Deep?cited by 466
  17. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Modelcited by 450
  18. 2025GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learningcited by 449
  19. 2015Policy Distillationcited by 414
  20. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  21. 2026Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Modelscited by 386
  22. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  23. 2026Reinforcement Learning via Self-Distillationcited by 285
  24. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 261
  25. 2024TextGrad: Automatic "Differentiation" via Textcited by 250
  26. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  27. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  28. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 212
  29. 2026Self-Distillation Enables Continual Learningcited by 212
  30. 2026MiMo-V2-Flash Technical Reportcited by 163
  31. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  32. 2023Chain of Hindsight Aligns Language Models with Feedbackcited by 147
  33. 2024Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agentscited by 144
  34. 2022Learning by Distilling Contextcited by 128
  35. 2023Training Language Models with Language Feedback at Scalecited by 125
  36. 2024Yell At Your Robot: Improving On-the-Fly from Language Correctionscited by 123
  37. 2025REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewardscited by 121
  38. 2024Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigationcited by 116

50 most cited of 93 All 93 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 49.

49 papers.

Sort:

By area:

  1. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?cited by 87
  2. 2026Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routingcited by 65
  3. 2026Embarrassingly Simple Self-Distillation Improves Code Generationcited by 36
  4. 2026Experiential Reinforcement Learningcited by 21
  5. 2026Aligning Language Models from User Interactionscited by 21
  6. 2026Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVRcited by 13
  7. 2026ECHO: Terminal Agents Learn World Models for Freecited by 13
  8. 2026Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Groundingcited by 9
  9. 2026OdysSim: Building Foundation Models for Human Behavior Simulationcited by 8
  10. 2026HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agentscited by 7
  11. 2026OPD+: Rethinking the Advantage Design for On-Policy Distillationcited by 7
  12. 2026Learning from Language Feedback via Variational Policy Distillationcited by 6
37 more
  1. 2026Complementary RL: Towards Efficient Experience-Driven Agent Learningcited by 5
  2. 2026Nudging Beyond the Comfort Zone: Efficient Strategy-Guided Exploration for RLVRcited by 4
  3. 2026Reinforcing Human Behavior Simulation via Verbal Feedbackcited by 4
  4. 2026The Role of Feedback Alignment in Self-Distillationcited by 4
  5. 2026RewardFlow: Topology-Aware Reward Propagation on State Graphs for Agentic RL with Large Language Modelscited by 3
  6. 2026Self-Improving 4D Perception via Self-Distillationcited by 3
  7. 2026FlexSQL: Flexible Exploration and Execution Make Better Text-to-SQL Agentscited by 3
  8. 2026AVSD: Adaptive-View Self-Distillation by Balancing Consensus and Teacher-Specific Privileged Signalscited by 3
  9. 2026SPIRAL: Learning to Search and Aggregatecited by 3
  10. 2026GEM-Occ: From Visual Geometry Evidence to Embodied Semantic Occupancy Memorycited by 3
  11. 2026Agent Explorative Policy Optimization for Multimodal Agentic Reasoningcited by 2
  12. 2026Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retrycited by 2
  13. 2026Learn from Whoever Is Right: Answer-Verified Multi-Teacher Distillation for Multi-Domain LLMscited by 2
  14. 2026Self-Distilled Reinforcement Learning for Co-Evolving Agentic Recommender Systemscited by 1
  15. 2026Self-Play Enhancement via Advantage-Weighted Refinement in Online Federated LLM Fine-Tuning with Real-Time Feedbackcited by 1
  16. 2026Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gapcited by 1
  17. 2026RL with Learnable Textual Feedback: A Bilevel Approachcited by 1
  18. 2026CORE: Contrastive Reflection Enables Rapid Improvements in Reasoningcited by 1
  19. 2026A Predictive Law for On-Policy Self-Distillation From World Feedbackcited by 1
  20. 2026Self-Trained Verification for Training- and Test-Time Self-Improvementcited by 1
  21. 2026Physics-Guided Policy Optimization with Self-Distillationcited by 1
  22. 2026Be My Tutor: On-Policy Co-Distillation for Mutual LLM Improvement via Peer Feedbackcited by 1
  23. 2026Rollout Pass-Rate Control: Steering Binary-Reward RL Toward Its Most Informative Regimecited by 0
  24. 2026EchoDistill: Robust Large Audio Language Models via Noisy-to-Clean Self-Distillationcited by 0
  25. 2026Connect the Dots: Training LLMs for Long-Lifecycle Agents with Cross-Domain Generalization Via Reinforcement Learningcited by 0
  26. 2026QVal: Cheaply Evaluating Dense Supervision Signals for Long-Horizon LLM Agentscited by 0
  27. 2026Test-Time Scaling via Error Localizationcited by 0
  28. 2026The Rise of Verbal Reinforcement Learningcited by 0
  29. 2026Efficient Test-Time Adaptation through Human-AI Interactioncited by 0
  30. 2026Few-Shot Demonstrations Elicit the Use of In-Context World Representations in LLMscited by 0
  31. 2026TISD: On-Policy Self-Distillation with Trajectory Interventioncited by 0
  32. 2026Inductive Feedback for Mixed-Policy Distillationcited by 0
  33. 2026Shockingly Simple Self-retrospection Improves Agentic Models Without RLcited by 0
  34. 2026RLTL;DR: Self-improvement by Internalizing Self-generated Feedbackcited by 0
  35. 2026RISED: RubrIcs for agentic multi-environment Selection and sElf-Distillationcited by 0
  36. 2026MIMESIS: Learning User Simulators as Training Environments for Interactive Agentscited by 0
  37. 2026Environmental Feedback Modeling Matters: Rethinking Feedback Treatment in Agentic Hindsight Self-Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Reinforcement Learning from Reflective Feedback (RLRF): Aligning and Improving LLMs via Fine-Grained Self-Reflection
  2. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  4. 2023Reinforcement Learning in the Era of LLMs: What is Essential? What is needed? An RL Perspective on RLHF, Prompting, and Beyond
  5. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
  6. 2025No Free Lunch: Rethinking Internal Feedback for LLM Reasoning
  7. 2025Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective
  8. 2024Teaching Large Language Models to Reason with Reinforcement Learning