mlsanityData to 14 Oct 2026

Hierarchical Soft Actor-Critic for Sparse-Reward Long-Horizon Reinforcement Learning

arXiv 2607.23726 ↗First posted 26 Jul 2026cs.RO cs.LGcited by 0

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Exploration in sparse-reward long-horizon tasks poses significant challenges for reinforcement learning. To address these challenges, we propose a two-level Hierarchical Reinforcement Learning (HRL) framework. The first level handles high-level strategic planning, while the low-level uses the continuous-control Soft Actor-Critic (SAC) algorithm, and they utilize entropy-regularized policy optimization. The proposed framework was trained and evaluated using the Search-and-Rescue-2 (SAR-2) dataset. HRL-SAC effectively addresses sparse-reward long-horizon search problems characterized by delayed rewards and continuous control, and its outperforming the flat SAC baseline reinforcement learning in terms of success rates, coverage efficiency, and convergence. These findings indicate that hierarchical entropy-regularized policies are a promising solution to tackle long-horizon sparse-reward reinforcement learning tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 15 of its 32 references that are in the map. First posted on arXiv 26 Jul 2026.

Builds on

15 of 32 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Reinforcement Learning for Infinite-Dimensional Systems24 Sep
  2. 2024Generation of Geodesics with Actor-Critic Reinforcement Learning to Predict Midpoints2 Jul

Other references · 13

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  4. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  5. 2016The Option-Critic Architecturecited by 448
  6. 2017FeUdal Networks for Hierarchical Reinforcement Learningcited by 337
  7. 2016Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivationcited by 286
  8. 2018Large-Scale Study of Curiosity-Driven Learningcited by 280
  9. 2021LASER: Learning a Latent Action Space for Efficient Reinforcement Learningcited by 32
  10. 2017Learning Multi-Level Hierarchies with Hindsightcited by 20
  11. 2025Actor-Critic for Continuous Action Chunks: A Reinforcement Learning Framework for Long-Horizon Robotic Manipulation with Sparse Rewardcited by 13
  12. 2020Off-policy Maximum Entropy Reinforcement Learning : Soft Actor-Critic with Advantage Weighted Mixture Policy(SAC-AWMP)cited by 5
1 more
  1. 2024Hierarchical Decision Making Based on Structural Information Principlescited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Effective Reinforcement Learning Control using Conservative Soft Actor-Critic
  2. 2025Average-Reward Soft Actor-Critic
  3. 2025Max-Entropy Reinforcement Learning with Flow Matching and A Case Study on LQR
  4. 2024MENTOR: Guiding Hierarchical Reinforcement Learning with Human Feedback and Dynamic Distance Constraint
  5. 2024Improving Actor-Critic Training with Steerable Action-Value Approximation Errors
  6. 2025Bidirectional Soft Actor-Critic: Leveraging Forward and Reverse KL Divergence for Efficient Reinforcement Learning
  7. 2024ETGL-DDPG: A Deep Deterministic Policy Gradient Algorithm for Sparse Reward Continuous Control
  8. 2026FERPO: Forward Entropy-Regularized Policy Optimization