When LLM Reward Design Fails: Diagnostic-Driven Refinement for Sparse Structured RL
Not on the map. Why?
Abstract (arXiv)
For sparse, structured reinforcement-learning tasks with semantic reward-function interfaces, LLM-generated reward shaping is better framed as debugging than one-shot generation. We study PPO-trained agents using MiniGrid as core evaluation and MuJoCo as boundary stress test. Our audit finds two dominant one-shot failure modes -- reward flooding and semantic/API misunderstanding -- plus a rarer weak-shaping case. We propose diagnostic-driven iterative refinement, where training diagnostics and a failure-mode taxonomy guide targeted reward-function revision. Refinement improves DoorKey-8x8 from 2.3% to 97.6% and KeyCorridor from 31.2% to 86.7% with high seed-to-seed variance. Controls show these gains are not from retrying or extra training: metrics-only re-prompting yields large drops, while a static-vocabulary control recovers much of the gap (87.6%; 70.7%), showing the taxonomy prompt is a major mechanism and dynamic labels provide only partially isolated incremental evidence. Budget-matched and Best-of-3 comparisons separate refinement from selection and training-time effects. Component-removal tests, sensitivity analyses, and an audit against author labels provide converging evidence for the debugging interpretation while revealing calibration limits. Continuous-control results show the boundary: success-based diagnostics can misfire in dense-reward locomotion, and return-trend feedback removes one false-positive mechanism without robust gains. The low-call protocol is a cost contrast with population-based reward search, not a benchmark comparison. In four crossed-variance-design environments, point estimates suggest larger gains when LLM reward-function variance dominates but bootstrap intervals are wide. The method is bounded to sparse structured tasks with reliable interfaces under PPO; fields like event_text may help, hurt, or be neutral.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 26 of its 36 references that are in the map. First posted on arXiv 27 May 2026.
Builds on
26 of 36 references.
References · 26
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2023Code Llama: Open Foundation Models for Codecited by 2,105
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
- 2016Concrete Problems in AI Safetycited by 1,582
- 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
- 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
- 2018Exploration by Random Network Distillationcited by 966
- 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
- 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
- 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
- 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
14 more
- 2023Reward Design with Language Modelscited by 257
- 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
- 2020First return, then explorecited by 256
- 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
- 2022Defining and Characterizing Reward Hackingcited by 184
- 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 153
- 2024RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedbackcited by 147
- 2020Learning to Utilize Shaping Rewards: A New Approach of Reward Shapingcited by 112
- 2023Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraftcited by 60
- 2021On Bonus-Based Exploration Methods in the Arcade Learning Environmentcited by 37
- 2024A Large Language Model-Driven Reward Design Framework via Dynamic Feedback for Reinforcement Learningcited by 29
- 2026Enhancing Financial Report Question-Answering: A Retrieval-Augmented Generation System with Reranking Analysiscited by 20
- 2025AutoNeural: Co-Designing Vision-Language Models for NPU Inferencecited by 17
- 2026Sustainable Hybrid Document-Routed Retrieval for Financial RAG: Resolving the Robustness-Precision Trade-offcited by 13
Built on by
Citing papers per year: 2026: 1.
1 paper.