# CTSAC: Curriculum-Based Transformer Soft Actor-Critic for Goal-Oriented Robot Exploration

arXiv: https://arxiv.org/abs/2503.14254 · HTML: https://mlsanity.com/paper/2503.14254/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 18 of its 36 references that are in the map.
- First posted on arXiv 18 Mar 2025.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey](https://mlsanity.com/paper/2003.04960/) (2020)
- [Stabilizing Transformers for Reinforcement Learning](https://mlsanity.com/paper/1910.06764/) (2019)
- [Curriculum Learning: A Survey](https://mlsanity.com/paper/2101.10382/) (2021)
- [Cognitive Mapping and Planning for Visual Navigation](https://mlsanity.com/paper/1702.03920/) (2017)
- [Learning Exploration Policies for Navigation](https://mlsanity.com/paper/1903.01959/) (2019)
- [Virtual-to-real Deep Reinforcement Learning: Continuous Control of Mobile Robots for Mapless Navigation](https://mlsanity.com/paper/1703.00420/) (2017)
- [Drive Anywhere: Generalizable End-to-end Autonomous Driving with Multi-modal Foundation Models](https://mlsanity.com/paper/2310.17642/) (2023)
- [On Transforming Reinforcement Learning by Transformer: The Development Trajectory](https://mlsanity.com/paper/2212.14164/) (2022)
- [Goal-Driven Autonomous Exploration Through Deep Reinforcement Learning](https://mlsanity.com/paper/2103.07119/) (2021)
- [ARiADNE: A Reinforcement learning approach using Attention-based Deep Networks for Exploration](https://mlsanity.com/paper/2301.11575/) (2023)
- [FAR Planner: Fast, Attemptable Route Planner using Dynamic Visibility Update](https://mlsanity.com/paper/2110.09460/) (2021)
- [Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Rate](https://mlsanity.com/paper/2405.15384/) (2024)
- [MAexp: A Generic Platform for RL-based Multi-Agent Exploration](https://mlsanity.com/paper/2404.12824/) (2024)
- [Autonomous Exploration and Mapping for Mobile Robots via Cumulative Curriculum Reinforcement Learning](https://mlsanity.com/paper/2302.13025/) (2023)
- [Deep Reinforcement Learning for Autonomous Ground Vehicle Exploration Without A-Priori Maps](https://mlsanity.com/paper/2301.04036/) (2023)

## Built on by (most cited first, 1 of 1)

- [Accelerating Transformers in Online RL](https://mlsanity.com/paper/2509.26137/) (2025)

## Abstract (arXiv)

With the increasing demand for efficient and flexible robotic exploration solutions, Reinforcement Learning (RL) is becoming a promising approach in the field of autonomous robotic exploration. However, current RL-based exploration algorithms often face limited environmental reasoning capabilities, slow convergence rates, and substantial challenges in Sim-To-Real (S2R) transfer. To address these issues, we propose a Curriculum Learning-based Transformer Reinforcement Learning Algorithm (CTSAC) aimed at improving both exploration efficiency and transfer performance. To enhance the robot's reasoning ability, a Transformer is integrated into the perception network of the Soft Actor-Critic (SAC) framework, leveraging historical information to improve the farsightedness of the strategy. A periodic review-based curriculum learning is proposed, which enhances training efficiency while mitigating catastrophic forgetting during curriculum transitions. Training is conducted on the ROS-Gazebo continuous robotic simulation platform, with LiDAR clustering optimization to further reduce the S2R gap. Experimental results demonstrate the CTSAC algorithm outperforms the state-of-the-art non-learning and learning-based algorithms in terms of success rate and success rate-weighted exploration time. Moreover, real-world experiments validate the strong S2R transfer capabilities of CTSAC.