mlsanityData to 7 Oct 2026

From Reward Shaping to Q-Shaping: Achieving Unbiased Learning with LLM-Guided Knowledge

arXiv 2410.01458 ↗First posted 2 Oct 2024cs.AI cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Q-shaping is an extension of Q-value initialization and serves as an alternative to reward shaping for incorporating domain knowledge to accelerate agent training, thereby improving sample efficiency by directly shaping Q-values. This approach is both general and robust across diverse tasks, allowing for immediate impact assessment while guaranteeing optimality. We evaluated Q-shaping across 20 different environments using a large language model (LLM) as the heuristic provider. The results demonstrate that Q-shaping significantly enhances sample efficiency, achieving a 16.87% improvement over the best baseline in each environment and a 253.80% improvement compared to LLM-based reward shaping methods. These findings establish Q-shaping as a superior and unbiased alternative to conventional reward shaping in reinforcement learning.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 51 of its 61 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

51 of 61 references.

References · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2015Continuous control with deep reinforcement learningcited by 3,279
  5. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  7. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  8. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  9. 2016Generative Adversarial Imitation Learningcited by 1,576
  10. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  11. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  12. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
38 more
  1. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  2. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  3. 2024Yi: Open Foundation Models by 01.AIcited by 706
  4. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  5. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  6. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  7. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  8. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  9. 2023Language Models can Solve Computer Taskscited by 371
  10. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  11. 2022Transformers are Sample-Efficient World Modelscited by 317
  12. 2018Reward learning from human preferences and demonstrations in Ataricited by 314
  13. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  14. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
  15. 2023Reward Design with Language Modelscited by 257
  16. 2017Count-Based Exploration with Neural Density Modelscited by 255
  17. 2018Residual Reinforcement Learning for Robot Controlcited by 242
  18. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 230
  19. 2023Large Language Model Guided Tree-of-Thoughtcited by 218
  20. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 193
  21. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  22. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 173
  23. 2023SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Taskscited by 166
  24. 2015Incentivizing Exploration In Reinforcement Learning With Deep Predictive Modelscited by 154
  25. 2023Text2Reward: Reward Shaping with Language Models for Reinforcement Learningcited by 153
  26. 2022TransDreamer: Reinforcement Learning with Transformer World Modelscited by 143
  27. 2023Transformer-based World Models Are Happy With 100k Interactionscited by 137
  28. 2024Yell At Your Robot: Improving On-the-Fly from Language Correctionscited by 123
  29. 2023Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learningcited by 111
  30. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 96
  31. 2023From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfacescited by 67
  32. 2023Read and Reap the Rewards: Learning to Play Atari with the Help of Instruction Manualscited by 45
  33. 2022History Compression via Language Models in Reinforcement Learningcited by 39
  34. 2023Hierarchical Prompting Assists Large Language Model on Web Navigationcited by 36
  35. 2023Interactive Robot Learning from Verbal Correctioncited by 28
  36. 2023Accelerating Reinforcement Learning of Robotic Manipulations via Feedback from Large Language Modelscited by 22
  37. 2024Long-horizon Locomotion and Manipulation on a Quadrupedal Robot with Large Language Modelscited by 20
  38. 2023Semantic HELM: A Human-Readable Memory for Reinforcement Learningcited by 17

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Where You Go is Who You Are: Behavioral Theory-Guided LLMs for Inverse Reinforcement Learningcited by 0
  2. 2026medR: Reward Engineering for Clinical Offline Reinforcement Learning via Tri-Drive Potential Functionscited by 0
  3. 2026When Does External Guidance Help LLM Reasoning? A Bias-Variance Theory of Guidance-Augmented GRPOcited by 0