SAC-GLAM: Improving Online RL for LLM agents with Soft Actor-Critic and Hindsight Relabeling
On the map: LLM agents and tool use → Reinforcement learning for LLM agents
Abstract (arXiv)
The past years have seen Large Language Models (LLMs) strive not only as generative models but also as agents solving textual sequential decision-making tasks. When facing complex environments where their zero-shot abilities are insufficient, recent work showed online Reinforcement Learning (RL) could be used for the LLM agent to discover and learn efficient strategies interactively. However, most prior work sticks to on-policy algorithms, which greatly reduces the scope of methods such agents could use for both exploration and exploitation, such as experience replay and hindsight relabeling. Yet, such methods may be key for LLM learning agents, and in particular when designing autonomous intrinsically motivated agents sampling and pursuing their own goals (i.e. autotelic agents). This paper presents and studies an adaptation of Soft Actor-Critic and hindsight relabeling to LLM agents. Our method not only paves the path towards autotelic LLM agents that learn online but can also outperform on-policy methods in more classic multi-goal RL environments.
Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 15 of its 20 references that are in the map. First posted on arXiv 16 Oct 2024.
Builds on
15 of 20 references.
Closest ancestors · 3
Same direction, newest first
- 2024Reinforcing Language Agents via Policy Optimization with Action Decomposition23 May
- 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models10 Nov
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Models25 May
Other references · 12
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
- 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
- 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
- 2017Hindsight Experience Replaycited by 1,096
- 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
- 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
- 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 193
- 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
- 2019Soft Actor-Critic for Discrete Action Settingscited by 161
- 2020Autotelic Agents with Intrinsically Motivated Goal-Conditioned Reinforcement Learning: a Short Surveycited by 83
- 2020Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Explorationcited by 68
Built on by
Citing papers per year: 2025: 3; 2026: 2.
5 papers.
- 2025MAGELLAN: Metacognitive predictions of learning progress guide autotelic LLM agents in large goal spacescited by 7
- 2026Learning Abstractions for Hierarchical Planning in Program-Synthesis Agentscited by 3
- 2026COVR:Collaborative Optimization of VLMs and RL Agent for Visual-Based Controlcited by 1
- 2025Alchemist: Towards the Design of Efficient Online Continual Learning Systemcited by 0
- 2025Goal-Guided Efficient Exploration via Large Language Model in Reinforcement Learningcited by 0
- 2026Learning Abstractions for Hierarchical Planning in Program-Synthesis Agents31 Jan 2026
- 2026COVR:Collaborative Optimization of VLMs and RL Agent for Visual-Based Control4 Jan 2026
- 2025Goal-Guided Efficient Exploration via Large Language Model in Reinforcement Learning26 Sep 2025
- 2025Alchemist: Towards the Design of Efficient Online Continual Learning System3 Mar 2025
- 2025MAGELLAN: Metacognitive predictions of learning progress guide autotelic LLM agents in large goal spaces11 Feb 2025
Similar, not cited
Close in wording, no citation either way
- 2026Rethinking Agentic Reinforcement Learning In Large Language Models
- 2026Hybrid LLM-Augmented Reinforcement Learning Agents for Complex Sequential Decision Tasks
- 2025Toward Efficient Exploration by Large Language Model Agents
- 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning
- 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Survey
- 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL
- 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Models
- 2026A Subgoal-driven Framework for Improving Long-Horizon LLM Agents