# A Survey On Enhancing Reinforcement Learning in Complex Environments: Insights from Human and LLM Feedback

arXiv: https://arxiv.org/abs/2411.13410 · HTML: https://mlsanity.com/paper/2411.13410/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 83 references that are in the map.
- First posted on arXiv 20 Nov 2024.

## Builds on (most cited first, 30 of 39)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [A Survey of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2312.14925/) (2023)
- [Reward Design with Language Models](https://mlsanity.com/paper/2303.00001/) (2023)
- [Guiding Pretraining in Reinforcement Learning with Large Language Models](https://mlsanity.com/paper/2302.06692/) (2023)
- [LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent](https://mlsanity.com/paper/2309.12311/) (2023)
- [Assessing Generalization in Deep Reinforcement Learning](https://mlsanity.com/paper/1810.12282/) (2018)
- [Yell At Your Robot: Improving On-the-Fly from Language Corrections](https://mlsanity.com/paper/2403.12910/) (2024)
- [WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment](https://mlsanity.com/paper/2402.12275/) (2024)
- [Motif: Intrinsic Motivation from Artificial Intelligence Feedback](https://mlsanity.com/paper/2310.00166/) (2023)
- [Language Understanding for Text-based Games Using Deep Reinforcement Learning](https://mlsanity.com/paper/1506.08941/) (2015)
- [Correcting Robot Plans with Natural Language Feedback](https://mlsanity.com/paper/2204.05186/) (2022)
- [Language Instructed Reinforcement Learning for Human-AI Coordination](https://mlsanity.com/paper/2304.07297/) (2023)
- [Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text](https://mlsanity.com/paper/2005.09382/) (2020)
- [Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation](https://mlsanity.com/paper/2006.14804/) (2020)
- [Ask Your Humans: Using Human Instructions to Improve Generalization in Reinforcement Learning](https://mlsanity.com/paper/2011.00517/) (2020)
- [HAIM-DRL: Enhanced Human-in-the-loop Reinforcement Learning for Safe and Efficient Autonomous Driving](https://mlsanity.com/paper/2401.03160/) (2024)
- [Towards End-to-End Learning for Dialog State Tracking and Management using Deep Reinforcement Learning](https://mlsanity.com/paper/1606.02560/) (2016)
- [Enabling Intelligent Interactions between an Agent and an LLM: A Reinforcement Learning Approach](https://mlsanity.com/paper/2306.03604/) (2023)
- [Beating Atari with Natural Language Guided Reinforcement Learning](https://mlsanity.com/paper/1704.05539/) (2017)
- [The Expertise Problem: Learning from Specialized Feedback](https://mlsanity.com/paper/2211.06519/) (2022)
- [Natural Language Reinforcement Learning](https://mlsanity.com/paper/2402.07157/) (2024)
- [Machine versus Human Attention in Deep Reinforcement Learning Tasks](https://mlsanity.com/paper/2010.15942/) (2020)
- [The RL/LLM Taxonomy Tree: Reviewing Synergies Between Reinforcement Learning and Large Language Models](https://mlsanity.com/paper/2402.01874/) (2024)
- [Instruction-Following Agents with Multimodal Transformer](https://mlsanity.com/paper/2210.13431/) (2022)
- [MLANet: Multi-Level Attention Network with Sub-instruction for Continuous Vision-and-Language Navigation](https://mlsanity.com/paper/2303.01396/) (2023)
- [FollowNet: Robot Navigation by Following Natural Language Directions with Deep Reinforcement Learning](https://mlsanity.com/paper/1805.06150/) (2018)
- [LLM Augmented Hierarchical Agents](https://mlsanity.com/paper/2311.05596/) (2023)
- [Exploiting Contextual Structure to Generate Useful Auxiliary Tasks](https://mlsanity.com/paper/2303.05038/) (2023)

## Built on by (most cited first, 9 of 9)

- [Large Language Model Agent: A Survey on Methodology, Applications and Challenges](https://mlsanity.com/paper/2503.21460/) (2025)
- [From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discovery](https://mlsanity.com/paper/2508.14111/) (2025)
- [Large language models for artificial general intelligence (AGI): A survey of foundational principles and approaches](https://mlsanity.com/paper/2501.03151/) (2025)
- [Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysis](https://mlsanity.com/paper/2601.20103/) (2026)
- [MEMTRACK: Evaluating Long-Term Memory and State Tracking in Multi-Platform Dynamic Agent Environments](https://mlsanity.com/paper/2510.01353/) (2025)
- [Evaluation of LLMs for mathematical problem solving](https://mlsanity.com/paper/2506.00309/) (2025)
- [LLM-Guided Reinforcement Learning: Addressing Training Bottlenecks through Policy Modulation](https://mlsanity.com/paper/2505.20671/) (2025)
- [Cleansing the Artificial Mind: A Self-Reflective Detoxification Framework for Large Language Models](https://mlsanity.com/paper/2601.11776/) (2026)
- [MARCO: Meta-Reflection with Cross-Referencing for Code Reasoning](https://mlsanity.com/paper/2505.17481/) (2025)

## Abstract (arXiv)

Reinforcement learning (RL) is one of the active fields in machine learning, demonstrating remarkable potential in tackling real-world challenges. Despite its promising prospects, this methodology has encountered with issues and challenges, hindering it from achieving the best performance. In particular, these approaches lack decent performance when navigating environments and solving tasks with large observation space, often resulting in sample-inefficiency and prolonged learning times. This issue, commonly referred to as the curse of dimensionality, complicates decision-making for RL agents, necessitating a careful balance between attention and decision-making. RL agents, when augmented with human or large language models' (LLMs) feedback, may exhibit resilience and adaptability, leading to enhanced performance and accelerated learning. Such feedback, conveyed through various modalities or granularities including natural language, serves as a guide for RL agents, aiding them in discerning relevant environmental cues and optimizing decision-making processes. In this survey paper, we mainly focus on problems of two-folds: firstly, we focus on humans or an LLMs assistance, investigating the ways in which these entities may collaborate with the RL agent in order to foster optimal behavior and expedite learning; secondly, we delve into the research papers dedicated to addressing the intricacies of environments characterized by large observation space.