mlsanityData to 7 Oct 2026

Bootstrapping Exploration with Group-Level Natural Language Feedback in Reinforcement Learning

arXiv 2603.04597 ↗First posted 4 Mar 2026cs.CL cs.AIcited by 5

On the map: Synthetic data and self-improvement for LLMs → Self-rewarding alignment and self-training

Abstract (arXiv)

Large language models (LLMs) typically receive diverse natural language (NL) feedback through interaction with the environment. However, current reinforcement learning (RL) algorithms rely solely on scalar rewards, leaving the rich information in NL feedback underutilized and leading to inefficient exploration. In this work, we propose GOLF, an RL framework that explicitly exploits group-level language feedback to guide targeted exploration through actionable refinements. GOLF aggregates two complementary feedback sources: (i) external critiques that pinpoint errors or propose targeted fixes, and (ii) intra-group attempts that supply alternative partial ideas and diverse failure patterns. These group-level feedbacks are aggregated to produce high-quality refinements, which are adaptively injected into training as off-policy scaffolds to provide targeted guidance in sparse-reward regions. Meanwhile, GOLF jointly optimizes generation and refinement within a unified RL loop, creating a virtuous cycle that continuously improves both capabilities. Experiments on both verifiable and non-verifiable benchmarks show that GOLF achieves superior performance and exploration efficiency, achieving 2.2× improvements in sample efficiency compared to RL methods trained solely on scalar rewards. Code is available at https://github.com/LuckyyySTA/GOLF.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 31 of its 44 references that are in the map. First posted on arXiv 4 Mar 2026.

Builds on

31 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025LANPO: Bootstrapping Language and Numerical Feedback for Reinforcement Learning in LLMs18 Oct

Other references · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  9. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  10. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  11. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  12. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 948
18 more
  1. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  2. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
  3. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  4. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 484
  5. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 328
  6. 2026Reinforcement Learning via Self-Distillationcited by 285
  7. 2025Learning to Reason under Off-Policy Guidancecited by 266
  8. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  9. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  10. 2025Generalizing Verifiable Instruction Followingcited by 178
  11. 2024Critique-out-Loud Reward Modelscited by 105
  12. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedbackcited by 86
  13. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 79
  14. 2025Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitatecited by 61
  15. 2025Language Models that Think, Chat Bettercited by 23
  16. 2025User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signalcited by 19
  17. 2025Text2Grad: Reinforcement Learning from Natural Language Feedbackcited by 18
  18. 2025RefineBench: Evaluating Refinement Capability of Language Models via Checklistscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026LANG: Reinforcement Learning for Multilingual Reasoning with Language-Adaptive Hint Guidancecited by 2
  2. 2026LLM-as-a-Coach: Experiential Learning for Non-Verifiable Taskscited by 2
  3. 2026From Memorization to Absorption: Mixed-Policy RL for Continual Knowledge Injectioncited by 0
  4. 2026Learning to Coach for Experiential Learningcited by 0
  5. 2026Thinking Leakage: A Causal Audit of NoThink Post-Training in Hybrid Reasoning Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Training
  2. 2026Expanding LLM Agent Boundaries with Strategy-Guided Exploration
  3. 2025Toward Efficient Exploration by Large Language Model Agents
  4. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs
  5. 2025Meta-RL Induces Exploration in Language Agents
  6. 2026TGRL: Temperature-Grouped Reinforcement Learning for Efficient Exploration in LLMs
  7. 2026MAGE: Meta-Reinforcement Learning for Language Agents toward Strategic Exploration and Exploitation
  8. 2026Prompt-Driven Exploration: Language as an Exploration Space for VLA Reinforcement Learning