# LgTS: Dynamic Task Sampling using LLM-generated sub-goals for Reinforcement Learning Agents

arXiv: https://arxiv.org/abs/2310.09454 · HTML: https://mlsanity.com/paper/2310.09454/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 14 Oct 2023.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)
- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 26 of 26)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Instruction Tuning with GPT-4](https://mlsanity.com/paper/2304.03277/) (2023)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [LLM+P: Empowering Large Language Models with Optimal Planning Proficiency](https://mlsanity.com/paper/2304.11477/) (2023)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models](https://mlsanity.com/paper/2212.04088/) (2022)
- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](https://mlsanity.com/paper/2003.04960/) (2020)
- [Reward Design with Language Models](https://mlsanity.com/paper/2303.00001/) (2023)
- [Teacher-Student Curriculum Learning](https://mlsanity.com/paper/1707.00183/) (2017)
- [Reward Machines: Exploiting Reward Function Structure in Reinforcement Learning](https://mlsanity.com/paper/2010.03950/) (2020)
- [Open-World Object Manipulation using Pre-trained Vision-Language Models](https://mlsanity.com/paper/2303.00905/) (2023)
- [Task and Motion Planning with Large Language Models for Object Rearrangement](https://mlsanity.com/paper/2303.06247/) (2023)
- [LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent](https://mlsanity.com/paper/2309.12311/) (2023)
- [Reinforcement Learning With Temporal Logic Rewards](https://mlsanity.com/paper/1612.03471/) (2016)
- [Control Synthesis from Linear Temporal Logic Specifications using Model-Free Reinforcement Learning](https://mlsanity.com/paper/1909.07299/) (2019)
- [Foundations for Restraining Bolts: Reinforcement Learning with LTLf/LDLf restraining specifications](https://mlsanity.com/paper/1807.06333/) (2018)
- [Transfer of Temporal Logic Formulas in Reinforcement Learning](https://mlsanity.com/paper/1909.04256/) (2019)
- [Multi-Agent Reinforcement Learning with Temporal Logic Specifications](https://mlsanity.com/paper/2102.00582/) (2021)
- [Overcoming Exploration: Deep Reinforcement Learning for Continuous Control in Cluttered Environments from Temporal Logic Specifications](https://mlsanity.com/paper/2201.12231/) (2022)
- [ACuTE: Automatic Curriculum Transfer from Simple to Complex Environments](https://mlsanity.com/paper/2204.04823/) (2022)

## Built on by (most cited first, 11 of 11)

- [LLM-Empowered State Representation for Reinforcement Learning](https://mlsanity.com/paper/2407.13237/) (2024)
- [Monte Carlo Planning with Large Language Model for Text-Based Game Agents](https://mlsanity.com/paper/2504.16855/) (2025)
- [The Evolving Landscape of LLM- and VLM-Integrated Reinforcement Learning](https://mlsanity.com/paper/2502.15214/) (2025)
- [Stop Wandering, Find the Keys: LLMs Discriminate Key States for Efficient Multi-Agent Exploration](https://mlsanity.com/paper/2410.02511/) (2024)
- [Creative Problem Solving in Large Language and Vision Models -- What Would it Take?](https://mlsanity.com/paper/2405.01453/) (2024)
- [Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making](https://mlsanity.com/paper/2506.02522/) (2025)
- [LLMDR: LLM-Driven Deadlock Detection and Resolution in Multi-Agent Pathfinding](https://mlsanity.com/paper/2503.00717/) (2025)
- [Agentic Episodic Control](https://mlsanity.com/paper/2506.01442/) (2025)
- [Novelty Adaptation Through Hybrid Large Language Model (LLM)-Symbolic Planning and LLM-guided Reinforcement Learning](https://mlsanity.com/paper/2603.11351/) (2026)
- [Goal-Guided Efficient Exploration via Large Language Model in Reinforcement Learning](https://mlsanity.com/paper/2509.22008/) (2025)
- [PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learning](https://mlsanity.com/paper/2609.31855/) (2026)

## Abstract (arXiv)

Recent advancements in reasoning abilities of Large Language Models (LLM) has promoted their usage in problems that require high-level planning for robots and artificial agents. However, current techniques that utilize LLMs for such planning tasks make certain key assumptions such as, access to datasets that permit finetuning, meticulously engineered prompts that only provide relevant and essential information to the LLM, and most importantly, a deterministic approach to allow execution of the LLM responses either in the form of existing policies or plan operators. In this work, we propose LgTS (LLM-guided Teacher-Student learning), a novel approach that explores the planning abilities of LLMs to provide a graphical representation of the sub-goals to a reinforcement learning (RL) agent that does not have access to the transition dynamics of the environment. The RL agent uses Teacher-Student learning algorithm to learn a set of successful policies for reaching the goal state from the start state while simultaneously minimizing the number of environmental interactions. Unlike previous methods that utilize LLMs, our approach does not assume access to a propreitary or a fine-tuned LLM, nor does it require pre-trained policies that achieve the sub-goals proposed by the LLM. Through experiments on a gridworld based DoorKey domain and a search-and-rescue inspired domain, we show that generating a graphical structure of sub-goals helps in learning policies for the LLM proposed sub-goals and the Teacher-Student learning algorithm minimizes the number of environment interactions when the transition dynamics are unknown.