mlsanityData to 7 Oct 2026

ReMA: Learning to Meta-think for LLMs with Multi-Agent Reinforcement Learning

arXiv 2503.09501 ↗First posted 12 Mar 2025cs.AI cs.CL cs.LGcited by 78

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

Recent research on Reasoning of Large Language Models (LLMs) has sought to further enhance their performance by integrating meta-thinking -- enabling models to monitor, evaluate, and control their reasoning processes for more adaptive and effective problem-solving. However, current single-agent work lacks a specialized design for acquiring meta-thinking, resulting in low efficacy. To address this challenge, we introduce Reinforced Meta-thinking Agents (ReMA), a novel framework that leverages Multi-Agent Reinforcement Learning (MARL) to elicit meta-thinking behaviors, encouraging LLMs to think about thinking. ReMA decouples the reasoning process into two hierarchical agents: a high-level meta-thinking agent responsible for generating strategic oversight and plans, and a low-level reasoning agent for detailed executions. Through iterative reinforcement learning with aligned objectives, these agents explore and learn collaboration, leading to improved generalization and robustness. Empirical results from single-turn experiments demonstrate that ReMA outperforms single-agent RL baselines on complex reasoning tasks, including competitive-level mathematical benchmarks and LLM-as-a-Judge benchmarks. Additionally, we further extend ReMA to multi-turn interaction settings, leveraging turn-level ratio and parameter sharing to improve efficiency. Comprehensive ablation studies further illustrate the evolving dynamics of each distinct agent, providing valuable insights into how the meta-thinking reasoning process enhances the reasoning capabilities of LLMs. Our code can be found in https://github.com/ziyuwan/ReMA-public

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 92 of its 117 references that are in the map. First posted on arXiv 12 Mar 2025.

Builds on

92 of 117 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning25 Feb
  2. 2025A Roadmap to Guide the Integration of LLMs in Hierarchical Planning14 Jan
  3. 2025Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains10 Jan
  4. 2024AFlow: Automating Agentic Workflow Generation14 Oct
  5. 2024Qwen2.5-Coder Technical Report18 Sep
  6. 2024Improving LLM Reasoning with Multi-Agent Tree-of-Thought Validator Agent17 Sep
  7. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate30 May

Other references · 85

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  11. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  12. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
38 more
  1. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  2. 2023Let's Verify Step by Stepcited by 3,598
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  5. 2015Trust Region Policy Optimizationcited by 2,503
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  7. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  8. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  9. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  10. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  11. 2025s1: Simple test-time scalingcited by 1,311
  12. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  13. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  14. 2022Red Teaming Language Models with Language Modelscited by 983
  15. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  16. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  17. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviorscited by 580
  18. 2025LIMO: Less is More for Reasoningcited by 488
  19. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  20. 2025Process Reinforcement through Implicit Rewardscited by 478
  21. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 459
  22. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  23. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  24. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 373
  25. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  26. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 311
  27. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  28. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  29. 2025Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Modelscited by 291
  30. 2023SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planningcited by 289
  31. 2022Generating Sequences by Learning to Self-Correctcited by 271
  32. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 260
  33. 2023Unleashing the Emergent Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaborationcited by 246
  34. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 222
  35. 2023Playing repeated games with Large Language Modelscited by 213
  36. 2023Evaluating the Performance of Large Language Models on GAOKAO Benchmarkcited by 182
  37. 2024Generative Reward Modelscited by 148
  38. 2024Marco-o1: Towards Open Reasoning Models for Open-Ended Solutionscited by 142

50 most cited of 85 All 85 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 43; 2026: 35.

78 papers.

Sort:

By area:

  1. 2025Memory in the Age of AI Agentscited by 285
  2. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  4. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  5. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  6. 2025SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learningcited by 89
  7. 2025SPICE: Self-Play In Corpus Environments Improves Reasoningcited by 78
  8. 2025MARFT: Multi-Agent Reinforcement Fine-Tuningcited by 51
  9. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  10. 2025The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary Giantscited by 37
  11. 2025Leanabell-Prover: Posttraining Scaling in Formal Reasoningcited by 27
  12. 2025CoMAS: Co-Evolving Multi-Agent Systems via Interaction Rewardscited by 27
38 more
  1. 2025Do We Truly Need So Many Samples? Multi-LLM Repeated Sampling Efficiently Scales Test-Time Computecited by 20
  2. 2025RingMo-Agent: A Unified Remote Sensing Foundation Model for Multi-Platform and Multi-Modal Reasoningcited by 19
  3. 2025Aegis: Automated Error Generation and Attribution for Multi-Agent Systemscited by 19
  4. 2025Plan Then Action:High-Level Planning Guidance Reinforcement Learning for LLM Reasoningcited by 17
  5. 2025Not All Thoughts are Generated Equal: Efficient LLM Reasoning via Multi-Turn Reinforcement Learningcited by 16
  6. 2026Auditing Multi-Agent LLM Reasoning Trees Outperforms Majority Vote and LLM-as-Judgecited by 16
  7. 2026MAGIC: A Co-Evolving Attacker-Defender Adversarial Game for Robust LLM Safetycited by 15
  8. 2026Dr. MAS: Stable Reinforcement Learning for Multi-Agent LLM Systemscited by 15
  9. 2025Maestro: Learning to Collaborate via Conditional Listwise Policy Optimization for Multi-Agent LLMscited by 14
  10. 2025a1: Steep Test-time Scaling Law via Environment Augmented Generationcited by 13
  11. 2025Reinforcement Learning Foundations for Deep Research Systems: A Surveycited by 13
  12. 2026Heterogeneous Agent Collaborative Reinforcement Learningcited by 13
  13. 2025A Survey on Collaborative Mechanisms Between Large and Small Language Modelscited by 12
  14. 2025MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent Systemscited by 12
  15. 2025How to Train a Leader: Hierarchical Reasoning in Multi-Agent LLMscited by 11
  16. 2025Unlocking the Power of Multi-Agent LLM for Reasoning: From Lazy Agents to Deliberationcited by 11
  17. 2025Meta-Thinking in LLMs via Multi-Agent Reinforcement Learning: A Surveycited by 10
  18. 2025Neural Brain: A Neuroscience-inspired Framework for Embodied Agentscited by 10
  19. 2025HiMA-Ecom: Enabling Joint Training of Hierarchical Multi-Agent E-commerce Assistantscited by 9
  20. 2026Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learningcited by 9
  21. 2026Memory-R2: Fair Credit Assignment for Long-Horizon Memory-Augmented LLM Agentscited by 9
  22. 2026WebAnchor: Anchoring Agent Planning to Stabilize Long-Horizon Web Reasoningcited by 7
  23. 2025VideoChat-M1: Collaborative Policy Planning for Video Understanding via Multi-Agent Reinforcement Learningcited by 6
  24. 2026SAGE: Multi-Agent Self-Evolution for LLM Reasoningcited by 6
  25. 2026SPADE: Self-Play in Adaptive Synthetic Executable Environmentscited by 6
  26. 2025Topological Structure Learning Should Be A Research Priority for LLM-Based Multi-Agent Systemscited by 5
  27. 2025Off-Trajectory Reasoning: Can LLMs Collaborate on Reasoning Trajectory?cited by 5
  28. 2025Generative Adversarial Reasoner: Enhancing LLM Reasoning with Adversarial Reinforcement Learningcited by 5
  29. 2025PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computingcited by 4
  30. 2025Evo-MARL: Co-Evolutionary Multi-Agent Reinforcement Learning for Internalized Safetycited by 4
  31. 2025StyleBench: Evaluating thinking styles in Large Language Modelscited by 4
  32. 2025Interactive Learning for LLM Reasoningcited by 4
  33. 2026CircuitLM: A Multi-Agent LLM-Aided Design Framework for Generating Circuit Schematics from Natural Language Promptscited by 4
  34. 2026The Alignment Tax: Response Homogenization in Aligned LLMs and Its Implications for Uncertainty Estimationcited by 4
  35. 2025Gradient Coupling: The Hidden Barrier to Generalization in Agentic Reinforcement Learningcited by 3
  36. 2025AdvEvo-MARL: Shaping Internalized Safety through Adversarial Co-Evolution in Multi-Agent Reinforcement Learningcited by 3
  37. 2025Verifying Meta-Awareness via Predictive Rewards in Reasoning Modelscited by 3
  38. 2025Searching Meta Reasoning Skeleton to Guide LLM Reasoningcited by 3

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026MAGE: Meta-Reinforcement Learning for Language Agents toward Strategic Exploration and Exploitation
  2. 2025Cog-Rethinker: Hierarchical Metacognitive Reinforcement Learning for LLM Reasoning
  3. 2026Rethinking Agentic Reinforcement Learning In Large Language Models
  4. 2025Reinforce LLM Reasoning through Multi-Agent Reflection
  5. 2025Learning to Deliberate: Meta-policy Collaboration for Agentic LLMs with Multi-agent Reinforcement Learning
  6. 2024SMART: Self-learning Meta-strategy Agent for Reasoning Tasks
  7. 2025Multi-Agent Evolve: LLM Self-Improve through Co-evolution
  8. 2025DRAFT-RL: Multi-Agent Chain-of-Draft Reasoning for Reinforcement Learning-Enhanced LLMs