mlsanityData to 7 Oct 2026

Policy-Invariant Reward Shaping from LLM Feedback: A Framework for Hybrid RL Agents

arXiv 2608.18008 ↗First posted 18 Aug 2026cs.LG cs.AIcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Combining large language models with reinforcement learning is increasingly explored, yet the theoretical status of LLM-derived reward signals is often left implicit. We formalize the hybrid LLM-planner and RL-controller architecture as a Goal-Augmented Markov Decision Process and show that when the LLM per-state progress score is used as a bounded potential function, the resulting shaping term preserves the optimal policy set even when the LLM scores are inaccurate. This guarantee is stronger than what general LLM-as-reward approaches provide. We verify the result numerically on a small MDP under four potential configurations, including an adversarial one scaled to twenty times the base reward magnitude.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 23 of its 25 references that are in the map. First posted on arXiv 18 Aug 2026.

Builds on

23 of 25 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learning20 Mar
  3. 2023Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modelling28 Jan

Other references · 20

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  4. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  5. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  6. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  7. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  8. 2022ProgPrompt: Generating Situated Robot Task Plans using Large Language Modelscited by 598
  9. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  10. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  11. 2023Reward Design with Language Modelscited by 257
  12. 2016Modular Multitask Reinforcement Learning with Policy Sketchescited by 200
8 more
  1. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 193
  2. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  3. 2023SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Taskscited by 166
  4. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 153
  5. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  6. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  7. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
  8. 2017Learning with Latent Languagecited by 75

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective
  2. 2026LLM-ALSO: LLM-Driven Adaptive Learning-Signal Optimization for Multi-Agent Reinforcement Learning
  3. 2026Hybrid LLM-Augmented Reinforcement Learning Agents for Complex Sequential Decision Tasks
  4. 2025Zero-Shot LLMs in Human-in-the-Loop RL: Replacing Human Feedback for Reward Shaping
  5. 2026Regime-Conditional Stabilisation of LLM-Augmented Cooperative Multi-Agent Reinforcement Learning
  6. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Survey
  7. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning
  8. 2026LLMs for High-Frequency Decision-Making: Normalized Action Reward-Guided Consistency Policy Optimization