mlsanityData to 14 Oct 2026

Efficient Soft Actor-Critic with LLM-Based Action-Level Guidance for Continuous Control

arXiv 2603.17468 ↗First posted 18 Mar 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

We present GuidedSAC, a novel reinforcement learning (RL) algorithm that facilitates efficient exploration in vast state-action spaces. GuidedSAC leverages large language models (LLMs) as intelligent supervisors that provide action-level guidance for the Soft Actor-Critic (SAC) algorithm. The LLM-based supervisor analyzes the most recent trajectory using state information and visual replays, offering action-level interventions that enable targeted exploration. Furthermore, we provide a theoretical analysis of GuidedSAC, proving that it preserves the convergence guarantees of SAC while improving convergence speed. Through experiments in both discrete and continuous control environments, including toy text tasks and complex MuJoCo benchmarks, we demonstrate that GuidedSAC consistently outperforms standard SAC and state-of-the-art exploration-enhanced variants (e.g., RND, ICM, and E3B) in terms of sample efficiency and final performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 30 references that are in the map. First posted on arXiv 18 Mar 2026.

Builds on

25 of 30 references.

References · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  6. 2018Exploration by Random Network Distillationcited by 993
  7. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 769
  8. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  9. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 571
  10. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  11. 2018Residual Reinforcement Learning for Robot Controlcited by 274
  12. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 189
13 more
  1. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 187
  2. 2018Episodic Curiosity through Reachabilitycited by 109
  3. 2020Sim-to-Real Learning of All Common Bipedal Gaits via Periodic Reward Compositioncited by 96
  4. 2024WoCoCo: Learning Whole-Body Humanoid Control with Sequential Contactscited by 61
  5. 2022Exploration via Elliptical Episodic Bonusescited by 51
  6. 2023RobotGPT: Robot Manipulation Learning from ChatGPTcited by 39
  7. 2024Efficient Reinforcement Learning with Large Language Model Priorscited by 30
  8. 2024Revisiting Reward Design and Evaluation for Robust Humanoid Standing and Walkingcited by 21
  9. 2024ExploRLLM: Guiding Exploration in Reinforcement Learning with Large Language Modelscited by 19
  10. 2025Learning from Active Human Involvement through Proxy Value Propagationcited by 19
  11. 2025Episodic Novelty Through Temporal Distancecited by 16
  12. 2018Intervention Aided Reinforcement Learning for Safe and Practical Policy Optimization in Navigationcited by 13
  13. 2024RLingua: Improving Reinforcement Learning Sample Efficiency in Robotic Manipulations With Large Language Modelscited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Guiding Exploration in Reinforcement Learning Through LLM-Augmented Observations
  2. 2025LLM-Guided Reinforcement Learning: Addressing Training Bottlenecks through Policy Modulation
  3. 2025Goal-Guided Efficient Exploration via Large Language Model in Reinforcement Learning
  4. 2025LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models
  5. 2025Natural Language Actor-Critic: Scalable Off-Policy Learning in Language Space
  6. 2026Learning Agentic Policy from Action Guidance
  7. 2026Expanding LLM Agent Boundaries with Strategy-Guided Exploration
  8. 2025Can LLMs Guide Their Own Exploration? Gradient-Guided Reinforcement Learning for LLM Reasoning