# AdaRefiner: Refining Decisions of Language Models with Adaptive Feedback

arXiv: https://arxiv.org/abs/2309.17176 · HTML: https://mlsanity.com/paper/2309.17176/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 46 references that are in the map.
- First posted on arXiv 29 Sep 2023.

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [Exploration by Random Network Distillation](https://mlsanity.com/paper/1810.12894/) (2018)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://mlsanity.com/paper/2303.16199/) (2023)
- [Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents](https://mlsanity.com/paper/2302.01560/) (2023)
- [Planning to Explore via Self-Supervised World Models](https://mlsanity.com/paper/2005.05960/) (2020)
- [MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback](https://mlsanity.com/paper/2309.10691/) (2023)
- [Language Conditioned Imitation Learning over Unstructured Data](https://mlsanity.com/paper/2005.07648/) (2020)
- [Benchmarking the Spectrum of Agent Capabilities](https://mlsanity.com/paper/2109.06780/) (2021)
- [AdaPlanner: Adaptive Planning from Feedback with Language Models](https://mlsanity.com/paper/2305.16653/) (2023)
- [Exploration in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2205.00824/) (2022)
- [Skill Induction and Planning with Latent Language](https://mlsanity.com/paper/2110.01517/) (2021)
- [Do Embodied Agents Dream of Pixelated Sheep: Embodied Decision Making using Language Guided World Modelling](https://mlsanity.com/paper/2301.12050/) (2023)
- [Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text](https://mlsanity.com/paper/2005.09382/) (2020)
- [LMPriors: Pre-Trained Language Models as Task-Specific Priors](https://mlsanity.com/paper/2210.12530/) (2022)
- [FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answering](https://mlsanity.com/paper/2308.12060/) (2023)
- [Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learning](https://mlsanity.com/paper/2101.07393/) (2021)

## Built on by (most cited first, 10 of 10)

- [VideoGameBunny: Towards vision assistants for video games](https://mlsanity.com/paper/2407.15295/) (2024)
- [A Large-Scale Simulation on Large Language Models for Decision-Making in Political Science](https://mlsanity.com/paper/2412.15291/) (2024)
- [The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learning](https://mlsanity.com/paper/2502.15214/) (2025)
- [CyberV: Cybernetics for Test-time Scaling in Video Understanding](https://mlsanity.com/paper/2506.07971/) (2025)
- [World Models with Hints of Large Language Models for Goal Achieving](https://mlsanity.com/paper/2406.07381/) (2024)
- [Causal-aware Large Language Models: Enhancing Decision-Making Through Learning, Adapting and Acting](https://mlsanity.com/paper/2505.24710/) (2025)
- [Expanding the Action Space of LLMs to Reason Beyond Language](https://mlsanity.com/paper/2510.07581/) (2025)
- [Subgoal Graph-Augmented Planning for LLM-Guided Open-World Reinforcement Learning](https://mlsanity.com/paper/2511.20993/) (2025)
- [Retrieval-Augmented Interpretable Learning: Towards Task-Specific Zero-Shot Models in Healthcare](https://mlsanity.com/paper/2607.17508/) (2026)
- [Goal-Guided Efficient Exploration via Large Language Model in Reinforcement Learning](https://mlsanity.com/paper/2509.22008/) (2025)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated significant success across various domains. However, their application in complex decision-making tasks frequently necessitates intricate prompt engineering or fine-tuning, leading to challenges in unseen downstream tasks and heavy demands on computational resources. Meanwhile, Reinforcement Learning (RL) has been recognized as effective in decision-making problems but struggles in environments with sparse rewards, such as open-world games. To overcome these challenges, we introduce AdaRefiner, a novel framework designed to enhance the synergy between LLMs and RL feedback. The key component of AdaRefiner is a lightweight Adapter Language Model (LM), which automatically refines task comprehension based on feedback from RL agents. This method mitigates the need for intricate prompt engineering and intensive LLM fine-tuning while maintaining the LLMs' generalization abilities and enhancing their decision-making capabilities in downstream tasks. Empirical evaluations of AdaRefiner on 22 diverse tasks within the open-world game Crafter have demonstrated its superior effectiveness, especially in guiding agents towards higher-level and common-sense skills. Our work makes contributions to the automatic self-refinement of LLMs with RL feedback, offering a more adaptable and efficient solution for complex decision-making problems.