mlsanityData to 14 Oct 2026

Text2Grad: Reinforcement Learning from Natural Language Feedback

arXiv 2505.22338 ↗First posted 28 May 2025cs.CL cs.AIcited by 18

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Traditional RLHF optimizes language models with coarse, scalar rewards that mask the fine-grained reasons behind success or failure, leading to slow and opaque learning. Recent work augments RL with textual critiques through prompting or reflection, improving interpretability but leaving model parameters untouched. We introduce Text2Grad, a reinforcement-learning paradigm that turns free-form textual feedback into span-level gradients. Given human (or programmatic) critiques, Text2Grad aligns each feedback phrase with the relevant token spans, converts these alignments into differentiable reward signals, and performs gradient updates that directly refine the offending portions of the model's policy. This yields precise, feedback-conditioned adjustments instead of global nudges. Text2Grad is realized through three components: (1) a high-quality feedback-annotation pipeline that pairs critiques with token spans; (2) a fine-grained reward model that predicts span-level reward on answers while generating explanatory critiques; and (3) a span-level policy optimizer that back-propagates natural-language gradients. Across summarization, code generation, and question answering, Text2Grad consistently surpasses scalar-reward RL and prompt-only baselines, providing both higher task metrics and richer interpretability. Our results suggest that natural-language feedback can serve not only as explanations, but also as actionable training signals for fine-grained alignment. The code for our method is available at https://github.com/microsoft/Text2Grad.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 46 of its 50 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

46 of 50 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025SCAR: Shapley Credit Assignment for More Efficient RLHF26 May
  2. 2025Reinforcement Learning from Human Feedback16 Apr
  3. 2024Self-Evolved Reward Learning for LLMs1 Nov
  4. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions3 Oct
  5. 2024Generative Reward Models2 Oct
  6. 2024A Critical Evaluation of AI Feedback for Aligning Large Language Models19 Feb
  7. 2024Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation14 Jan
  8. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling11 Jan
  9. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  10. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback27 Jul
  11. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  12. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
3 more
  1. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  2. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  3. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 31

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  9. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  11. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  12. 2023Let's Verify Step by Stepcited by 3,789
19 more
  1. 2021Program Synthesis with Large Language Modelscited by 3,637
  2. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  3. 2020Learning to summarize from human feedbackcited by 2,588
  4. 2024A Survey on LLM-as-a-Judgecited by 1,423
  5. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  6. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  7. 2024TextGrad: Automatic "Differentiation" via Textcited by 273
  8. 2023Can large language models provide useful feedback on research papers? A large-scale empirical analysiscited by 200
  9. 2024Large Language Model-Brained GUI Agents: A Surveycited by 162
  10. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 148
  11. 2023Training Language Models with Language Feedback at Scalecited by 128
  12. 2025KodCode: A Diverse, Challenging, and Verifiable Synthetic Dataset for Codingcited by 119
  13. 2024Feedback Loops With Language Models Drive In-Context Reward Hackingcited by 77
  14. 2023Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generationcited by 74
  15. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  16. 2025UFO2: The Desktop AgentOScited by 41
  17. 2024Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraintcited by 39
  18. 2024HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generationcited by 39
  19. 2024AllHands: Ask Me Anything on Large-scale Verbatim Feedback via Large Language Modelscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 18.

18 papers.

Sort:

By area:

  1. 2026Reinforcement Learning via Self-Distillationcited by 332
  2. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 53
  3. 2026Aligning Language Models from User Interactionscited by 28
  4. 2026BiPACE: Bisimulation-Guided Policy Optimization with Action Counterfactual Estimation for LLM Agentscited by 7
  5. 2026EvoIdeator: Evolving Scientific Ideas through Checklist-Grounded Reinforcement Learningcited by 6
  6. 2026Bootstrapping Exploration with Group-Level Natural Language Feedback in Reinforcement Learningcited by 5
  7. 2026CLR-voyance: Reinforcing Open-Ended Reasoning for Inpatient Clinical Decision Support with Outcome-Aware Rubricscited by 3
  8. 2026Inference-time Unlearning Using Conformal Predictioncited by 2
  9. 2026The Detection-Extraction Gap: Models Know the Answer Before They Can Say Itcited by 2
  10. 2026AdNanny: One Reasoning LLM for All Offline Ads Recommendation Taskscited by 1
  11. 2026ALIVE: Awakening LLM Reasoning via Adversarial Learning and Instructive Verbal Evaluationcited by 1
  12. 2026Introspective X Training: Feedback Conditioning Improves Scaling Across all LLM Training Stagescited by 1
6 more
  1. 2026RL with Learnable Textual Feedback: A Bilevel Approachcited by 1
  2. 2026Rubric-to-Code Credit Assignment for Reinforcement Learningcited by 1
  3. 2026What Is Missing: Interpretable Ratings for Large Language Model Outputscited by 0
  4. 2026The Rise of Verbal Reinforcement Learningcited by 0
  5. 2026DivOPD: Spread Wide, Look Close for Asynchronous On-Policy Distillation of Multi-turn Agentscited by 0
  6. 2026Training LLM Judges from Language Feedback via Position-Selective Self-Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
  2. 2023RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs
  3. 2026T5: Twin-Critic Training for Token-Level Thoughts in Reinforcement Mid-Training
  4. 2019Fine-Tuning Language Models from Human Preferences
  5. 2024Self-Generated Critiques Boost Reward Modeling for Language Models
  6. 2024Improving Reward Models with Synthetic Critiques
  7. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback
  8. 2025Critique-RL: Training Language Models for Critiquing through Two-Stage Reinforcement Learning