mlsanityData to 7 Oct 2026

Natural Language Reinforcement Learning

arXiv 2411.14251 ↗First posted 21 Nov 2024cs.LG cs.AI cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Artificial intelligence progresses towards the "Era of Experience," where agents are expected to learn from continuous, grounded interaction. We argue that traditional Reinforcement Learning (RL), which typically represents value as a scalar, can restrict agent's deep understanding of environments and hinders the active, deliberative learning crucial for navigating this new paradigm. To address the issue, we introduce Natural Language Reinforcement Learning (NLRL), a framework that extends RL principles into natural language counterparts. Central to NLRL is the Language Value Function (LVF), which redefines value as an interpretable linguistic narrative articulating the rationale behind an evaluation. NLRL further extends this concept to core RL components, including policy, the Bellman equation, and policy iteration. Leveraging recent advancements in Large Language Models (LLMs), NLRL can be practically implemented to achieve RL-like policy and value training through unsupervised environment interactions. Experiments over 4 multi-step agentic tasks demonstrate NLRL's effectiveness, efficiency, and its potential to foster deeper understanding and more active learning strategies.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 59 of its 77 references that are in the map. First posted on arXiv 21 Nov 2024.

Builds on

59 of 77 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2024The Llama 3 Herd of Modelscited by 14,192
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  8. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  9. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  10. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  11. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  12. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
38 more
  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  2. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  3. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  4. 2022Emergent Abilities of Large Language Modelscited by 2,483
  5. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  6. 2023AgentBench: Evaluating LLMs as Agentscited by 1,211
  7. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  8. 2023PaLM 2 Technical Reportcited by 1,154
  9. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  10. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  11. 2023Large Language Models as Optimizerscited by 830
  12. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 465
  13. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  14. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  15. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 311
  16. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  17. 2023Reward Design with Language Modelscited by 257
  18. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 256
  19. 2024TextGrad: Automatic "Differentiation" via Textcited by 250
  20. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  21. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 201
  22. 2024DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Searchcited by 189
  23. 2023Large Language Models for Generative Information Extraction: A Surveycited by 183
  24. 2019OpenSpiel: A Framework for Reinforcement Learning in Gamescited by 180
  25. 2023Generative Judge for Evaluating Alignmentcited by 163
  26. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 157
  27. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 155
  28. 2024Generative Reward Modelscited by 148
  29. 2023RoboCat: A Self-Improving Generalist Agent for Robotic Manipulationcited by 108
  30. 2016Model-Free Episodic Controlcited by 100
  31. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  32. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 84
  33. 2023ChessGPT: Bridging Policy Learning and Language Modelingcited by 73
  34. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
  35. 2023TIGERScore: Towards Building Explainable Metric for All Text Generation Taskscited by 69
  36. 2024Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMscited by 62
  37. 2022Retrieval-Augmented Reinforcement Learningcited by 53
  38. 2020Learning Rewards from Linguistic Feedbackcited by 41

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Cogito, Ergo Ludo: An Agent that Learns to Play by Reasoning and Planningcited by 12
  2. 2026DIVE: Unlocking Self-Improvement in Frozen Language Models Through Diversity-Driven Skill Evolutioncited by 2
  3. 2025Rule-Bottleneck Reinforcement Learning: Joint Explanation and Decision Optimization for Resource Allocation with Language Agentscited by 1
  4. 2026Can Jev be Your Q or Policy in Reinforcement Learning?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs
  2. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
  3. 2026NLPG: Natural-Language Policy Gradients for Self-Evolving Language Agents
  4. 2025PARL: Prompt-based Agents for Reinforcement Learning
  5. 2023Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Models
  6. 2026Rethinking Agentic Reinforcement Learning In Large Language Models
  7. 2025Natural Language Actor-Critic: Scalable Off-Policy Learning in Language Space
  8. 2024Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement