# Mutual Enhancement of Large Language and Reinforcement Learning Models through Bi-Directional Feedback Mechanisms: A Planning Case Study

arXiv: https://arxiv.org/abs/2401.06603 · HTML: https://mlsanity.com/paper/2401.06603/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 17 of its 24 references that are in the map.
- First posted on arXiv 12 Jan 2024.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 17 of 17)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [Prompting Large Language Model for Machine Translation: A Case Study](https://mlsanity.com/paper/2301.07069/) (2023)
- [Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning](https://mlsanity.com/paper/2302.02662/) (2023)
- [Guiding Pretraining in Reinforcement Learning with Large Language Models](https://mlsanity.com/paper/2302.06692/) (2023)
- [A Review of Safe Reinforcement Learning: Methods, Theory and Applications](https://mlsanity.com/paper/2205.10330/) (2022)
- [Language as an Abstraction for Hierarchical Deep Reinforcement Learning](https://mlsanity.com/paper/1906.07343/) (2019)
- [RL4F: Generating Natural Language Feedback with Reinforcement Learning for Repairing Model Outputs](https://mlsanity.com/paper/2305.08844/) (2023)
- [Large Language Models as Generalizable Policies for Embodied Tasks](https://mlsanity.com/paper/2310.17722/) (2023)
- [Survey on reinforcement learning for language processing](https://mlsanity.com/paper/2104.05565/) (2021)
- [Introspective Tips: Large Language Model for In-Context Decision Making](https://mlsanity.com/paper/2305.11598/) (2023)
- [Safe Reinforcement Learning with Natural Language Constraints](https://mlsanity.com/paper/2010.05150/) (2020)
- [A Human-Centered Safe Robot Reinforcement Learning Framework with Interactive Behaviors](https://mlsanity.com/paper/2302.13137/) (2023)
- [BabyStories: Can Reinforcement Learning Teach Baby Language Models to Write Better Stories?](https://mlsanity.com/paper/2310.16681/) (2023)

## Built on by (most cited first, 3 of 3)

- [A Survey On Enhancing Reinforcement Learning in Complex Environments: Insights from Human and LLM Feedback](https://mlsanity.com/paper/2411.13410/) (2024)
- [Long Context, Less Focus: A Scaling Gap in LLMs Revealed through Privacy and Personalization](https://mlsanity.com/paper/2602.15028/) (2026)
- [Federation Is Nearly Free, Reasoning Is Not: Tradeoffs for AI Co-Scientists in Protein Characterization Workflows](https://mlsanity.com/paper/2608.25215/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have demonstrated remarkable capabilities for reinforcement learning (RL) models, such as planning and reasoning capabilities. However, the problems of LLMs and RL model collaboration still need to be solved. In this study, we employ a teacher-student learning framework to tackle these problems, specifically by offering feedback for LLMs using RL models and providing high-level information for RL models with LLMs in a cooperative multi-agent setting. Within this framework, the LLM acts as a teacher, while the RL model acts as a student. The two agents cooperatively assist each other through a process of recursive help, such as "I help you help I help." The LLM agent supplies abstract information to the RL agent, enabling efficient exploration and policy improvement. In turn, the RL agent offers feedback to the LLM agent, providing valuable, real-time information that helps generate more useful tokens. This bi-directional feedback loop promotes optimization, exploration, and mutual improvement for both agents, enabling them to accomplish increasingly challenging tasks. Remarkably, we propose a practical algorithm to address the problem and conduct empirical experiments to evaluate the effectiveness of our method.