mlsanityData to 14 Oct 2026

MoL-RL: Distilling Multi-Step Environmental Feedback into LLMs for Feedback-Independent Reasoning

arXiv 2507.20278 ↗First posted 27 Jul 2025cs.CLcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Large language models (LLMs) face significant challenges in effectively leveraging sequential environmental feedback (EF) signals, such as natural language evaluations, for feedback-independent chain-of-thought (CoT) reasoning. Existing approaches either convert EF into scalar rewards, losing rich contextual information, or employ refinement datasets, failing to exploit the multi-step and discrete nature of EF interactions. To address these limitations, we propose MoL-RL, a novel training paradigm that integrates multi-step EF signals into LLMs through a dual-objective optimization framework. Our method combines MoL (Mixture-of-Losses) continual training, which decouples domain-specific EF signals (optimized via cross-entropy loss) and general language capabilities (preserved via Kullback-Leibler divergence), with GRPO-based post-training to distill sequential EF interactions into single-step inferences. This synergy enables robust feedback-independent reasoning without relying on external feedback loops. Experimental results on mathematical reasoning (MATH-500, AIME24/AIME25) and code generation (CodeAgent-Test) benchmarks demonstrate that MoL-RL achieves state-of-the-art performance with the Qwen3-8B model, while maintaining strong generalization across model scales (Qwen3-4B). This work provides a promising approach for leveraging multi-step textual feedback to enhance LLMs' reasoning capabilities in diverse domains.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 13 of its 24 references that are in the map. First posted on arXiv 27 Jul 2025.

Builds on

13 of 24 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitate29 Jan

Other references · 12

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2018World Modelscited by 1,513
  5. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  6. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 239
  7. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedbackcited by 90
  8. 2025J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learningcited by 75
  9. 2025ReCode: Updating Code API Knowledge with Reinforcement Learningcited by 12
  10. 2025From Reasoning to Code: GRPO Optimization for Underrepresented Languagescited by 11
  11. 2025MoL for LLMs: Dual-Loss Optimization to Enhance Domain Expertise While Preserving General Capabilitiescited by 3
  12. 2025LLM should think and action as a humancited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Teaching Large Language Models to Reason with Reinforcement Learning
  2. 2023MAF: Multi-Aspect Feedback for Improving Reasoning in Large Language Models
  3. 2026Correct, Concise and Complete: Multi-stage Training For Adaptive Reasoning
  4. 2025Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoning
  5. 2024Fine-Tuning on Diverse Reasoning Chains Drives Within-Inference CoT Refinement in LLMs
  6. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models
  7. 2025Post-Training Large Language Models via Reinforcement Learning from Self-Feedback
  8. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches