mlsanityData to 14 Oct 2026

The Rise of Verbal Reinforcement Learning

arXiv 2609.01597 ↗First posted 1 Sep 2026cs.CL cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

Natural language is emerging as a primary feedback channel for improving language agents, capable of conveying intent, preferences, and causal structure in forms interpretable by both humans and modern language models. We call this paradigm Verbal Reinforcement Learning (VRL) and offer the first unified account of it. We organize the field around a single axis, textitwhen verbal feedback takes effect in an agent's lifecycle and textitwhat it modifies, yielding three pillars: (1) Language as Grounding Signal, where language defines the task itself by specifying goals, states, and reward structures; (2) Language as Deliberative Feedback, where natural language guides reasoning at test time without the need to update model parameters; (3) Language as Learning Signal, where language-based feedback shapes model parameters through training. Within each pillar, we synthesize representative work, distinguish key subcategories of approaches, and outline the distinct role language plays in shaping agent behavior. Together, this taxonomy shows how verbal reinforcement is reshaping agent development, while also defining the challenges and opportunities for building more capable and aligned agents.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 84 of its 120 references that are in the map. First posted on arXiv 1 Sep 2026.

Builds on

84 of 120 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedback2 Feb
  2. 2025Text2Grad: Reinforcement Learning from Natural Language Feedback28 May

Other references · 82

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2023Let's Verify Step by Stepcited by 3,789
  9. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  10. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  12. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
38 more
  1. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  2. 2016Concrete Problems in AI Safetycited by 1,642
  3. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  4. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
  5. 2024The AI Scientist: Towards Fully Automated Open-Ended Scientific Discoverycited by 1,071
  6. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  7. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debatecited by 1,011
  8. 2023Teaching Large Language Models to Self-Debugcited by 988
  9. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  10. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  12. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  13. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
  14. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  15. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 783
  16. 2022Large Language Models Can Self-Improvecited by 664
  17. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
  18. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
  19. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  20. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  21. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  22. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  23. 2024Improve Mathematical Reasoning in Language Models by Automated Process Supervisioncited by 441
  24. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 356
  25. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  26. 2018TextWorld: A Learning Environment for Text-based Gamescited by 310
  27. 2024KTO: Model Alignment as Prospect Theoretic Optimizationcited by 264
  28. 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 244
  29. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 240
  30. 2023Is Self-Repair a Silver Bullet for Code Generation?cited by 197
  31. 2023Self-Evaluation Guided Beam Search for Reasoningcited by 196
  32. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 160
  33. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 158
  34. 2025Prompt Injection Attack to Tool Selection in LLM Agentscited by 158
  35. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  36. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 151
  37. 2024LLM Critics Help Catch LLM Bugscited by 147
  38. 2025Memory Injection Attacks on LLM Agents via Query-Only Interactioncited by 144

50 most cited of 82 All 82 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Natural Language Reinforcement Learning
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learning
  3. 2023Is Feedback All You Need? Leveraging Natural Language Feedback in Goal-Conditioned Reinforcement Learning
  4. 2024Teaching Embodied Reinforcement Learning Agents: Informativeness and Diversity of Language Use
  5. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Survey
  6. 2026Rethinking Agentic Reinforcement Learning In Large Language Models
  7. 2025Narrative-Guided Reinforcement Learning: A Platform for Studying Language Model Influence on Decision Making
  8. 2025SpeakRL: Synergizing Reasoning, Speaking, and Acting in Language Models with Reinforcement Learning