# Reward Guidance for Reinforcement Learning Tasks Based on Large Language Models: The LMGT Framework

arXiv: https://arxiv.org/abs/2409.04744 · HTML: https://mlsanity.com/paper/2409.04744/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 53 references that are in the map.
- First posted on arXiv 7 Sep 2024.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Curiosity-driven Exploration by Self-supervised Prediction](https://mlsanity.com/paper/1705.05363/) (2017)
- [Is Q-learning Provably Efficient?](https://mlsanity.com/paper/1807.03765/) (2018)
- [A Tutorial on Thompson Sampling](https://mlsanity.com/paper/1707.02038/) (2017)
- [Language Models Represent Space and Time](https://mlsanity.com/paper/2310.02207/) (2023)
- [Large-Scale Study of Curiosity-Driven Learning](https://mlsanity.com/paper/1808.04355/) (2018)
- [Guiding Pretraining in Reinforcement Learning with Large Language Models](https://mlsanity.com/paper/2302.06692/) (2023)
- [Understanding the impact of entropy on policy optimization](https://mlsanity.com/paper/1811.11214/) (2018)
- [Behavior From the Void: Unsupervised Active Pre-Training](https://mlsanity.com/paper/2103.04551/) (2021)
- [RUDDER: Return Decomposition for Delayed Rewards](https://mlsanity.com/paper/1806.07857/) (2018)
- [Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization](https://mlsanity.com/paper/2308.02151/) (2023)
- [RecSim: A Configurable Simulation Platform for Recommender Systems](https://mlsanity.com/paper/1909.04847/) (2019)
- [Housekeep: Tidying Virtual Households using Commonsense Reasoning](https://mlsanity.com/paper/2205.10712/) (2022)
- [Rank the Episodes: A Simple Approach for Exploration in Procedurally-Generated Environments](https://mlsanity.com/paper/2101.08152/) (2021)
- [Leveraging the Inductive Bias of Large Language Models for Abstract Textual Reasoning](https://mlsanity.com/paper/2110.02370/) (2021)
- [Large Language Models in the Workplace: A Case Study on Prompt Engineering for Job Type Classification](https://mlsanity.com/paper/2303.07142/) (2023)
- [A Provably Efficient Sample Collection Strategy for Reinforcement Learning](https://mlsanity.com/paper/2007.06437/) (2020)
- [Plan-Based Relaxed Reward Shaping for Goal-Directed Tasks](https://mlsanity.com/paper/2107.06661/) (2021)
- [CogniDual Framework: Self-Training Large Language Models within a Dual-System Theoretical Framework for Improving Cognitive Tasks](https://mlsanity.com/paper/2409.03381/) (2024)
- [BTBR: A Bayesian-Theory-Driven Probabilistic-Fuzzy Framework for Implicit Bias Removal in Large Language Models](https://mlsanity.com/paper/2408.10608/) (2024)

## Built on by (most cited first, 4 of 4)

- [SCALAR: Learning and Composing Skills through LLM Guided Symbolic Planning and Deep RL Grounding](https://mlsanity.com/paper/2603.09036/) (2026)
- [PRISM: Festina Lente Proactivity -- Risk-Sensitive, Uncertainty-Aware Deliberation for Proactive Agents](https://mlsanity.com/paper/2602.01532/) (2026)
- [medR: Reward Engineering for Clinical Offline Reinforcement Learning via Tri-Drive Potential Functions](https://mlsanity.com/paper/2602.03305/) (2026)
- [Reward-Zero: Language Embedding Driven Implicit Reward Mechanisms for Reinforcement Learning](https://mlsanity.com/paper/2603.09331/) (2026)

## Abstract (arXiv)

The inherent uncertainty in the environmental transition model of Reinforcement Learning (RL) necessitates a delicate balance between exploration and exploitation. This balance is crucial for optimizing computational resources to accurately estimate expected rewards for the agent. In scenarios with sparse rewards, such as robotic control systems, achieving this balance is particularly challenging. However, given that many environments possess extensive prior knowledge, learning from the ground up in such contexts may be redundant. To address this issue, we propose Language Model Guided reward Tuning (LMGT), a novel, sample-efficient framework. LMGT leverages the comprehensive prior knowledge embedded in Large Language Models (LLMs) and their proficiency in processing non-standard data forms, such as wiki tutorials. By utilizing LLM-guided reward shifts, LMGT adeptly balances exploration and exploitation, thereby guiding the agent's exploratory behavior and enhancing sample efficiency. We have rigorously evaluated LMGT across various RL tasks and evaluated it in the embodied robotic environment Housekeep. Our results demonstrate that LMGT consistently outperforms baseline methods. Furthermore, the findings suggest that our framework can substantially reduce the computational resources required during the RL training phase.