mlsanityData to 14 Oct 2026

RL with Learnable Textual Feedback: A Bilevel Approach

arXiv 2605.24547 ↗First posted 23 May 2026cs.LGcited by 1

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning with verifiable rewards can improve LLM reasoning, but learning remains sample-inefficient when terminal rewards are sparse. This has motivated a growing line of work on RL with textual feedback, where a critic model generates natural language feedback to guide a reasoning model (the actor), augmenting scalar rewards with richer learning signals. However, existing methods typically treat feedback as fixed or auxiliary, which misses a key property: feedback should not merely be correct, but should improve the policy (actor model) when provided in context. This motivates a paradigm of learnable textual feedback for RL. Yet the learnability and usefulness of feedback depend on the policy's ability to learn from it, making RL with learnable feedback an inherently bilevel problem. We formalize this coupling as a Stackelberg bilevel program and derive Bilevel Natural Language Actor-Critic (Bi-NAC), which jointly trains a critic to generate reward-improving feedback and an actor to exploit it. Across MATH-500, MBPP, and GPQA, Bi-NAC improves sample and parameter efficiency over RL and fixed-critic baselines: our 2B model outperforms the 3B GRPO baseline, achieving 46.6% versus 41.4% on MATH-500, while our 6B model surpasses the 7B GRPO baseline, achieving 49.3% versus 43.6% on GPQA.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 26 of its 30 references that are in the map. First posted on arXiv 23 May 2026.

Builds on

26 of 30 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025From Faithfulness to Correctness: Generative Reward Models that Think Critically29 Sep
  2. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs17 Jun
  3. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  4. 2023Let's Verify Step by Step31 May
  5. 2022Solving math word problems with process- and outcome-based feedback25 Nov

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  7. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  8. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  9. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  10. 2018Behavioral Cloning from Observationcited by 492
  11. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  12. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 405
9 more
  1. 2022Self-critiquing models for assisting human evaluatorscited by 380
  2. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
  3. 2023Training Language Models with Language Feedback at Scalecited by 128
  4. 2022BOME! Bilevel Optimization Made Easy: A Simple First-Order Approachcited by 105
  5. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 53
  6. 2024Improving Reward Models with Synthetic Critiquescited by 44
  7. 2025Text2Grad: Reinforcement Learning from Natural Language Feedbackcited by 18
  8. 2026Reward Modeling from Natural Language Human Feedbackcited by 11
  9. 2024Towards Aligning Language Models with Textual Feedbackcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Beyond Parallel Sampling: Diverse Query Initialization for Agentic Searchcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025No Free Lunch: Rethinking Internal Feedback for LLM Reasoning
  2. 2025Critique-GRPO: Advancing LLM Reasoning with Natural Language and Numerical Feedback
  3. 2025Critique-RL: Training Language Models for Critiquing through Two-Stage Reinforcement Learning
  4. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection
  5. 2024Teaching Large Language Models to Reason with Reinforcement Learning
  6. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
  7. 2026Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning
  8. 2025RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments