mlsanityData to 7 Oct 2026

SALE-Based Offline Reinforcement Learning with Ensemble Q-Networks

arXiv 2501.03676 ↗First posted 7 Jan 2025cs.LG cs.AIcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In this work, we build upon the offline reinforcement learning algorithm TD7, which incorporates State-Action Learned Embeddings (SALE) and a prioritized experience replay buffer (LAP). We propose a model-free actor-critic algorithm that integrates ensemble Q-networks and a gradient diversity penalty from EDAC. The ensemble Q-networks introduce penalties to guide the actor network toward in-distribution actions, effectively addressing the challenge of out-of-distribution actions. Meanwhile, the gradient diversity penalty encourages diverse Q-value gradients, further suppressing overestimation for out-of-distribution actions. Additionally, our method retains an adjustable behavior cloning (BC) term that directs the actor network toward dataset actions during early training stages, while gradually reducing its influence as the precision of the Q-ensemble improves. These enhancements work synergistically to improve the stability and precision of the training. Experimental results on the D4RL MuJoCo benchmarks demonstrate that our algorithm achieves higher convergence speed, stability, and performance compared to existing methods.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 21 of its 29 references that are in the map. First posted on arXiv 7 Jan 2025.

Builds on

21 of 29 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learning26 Mar
  2. 2023Anti-Exploration by Random Network Distillation31 Jan
  3. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  4. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  5. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  6. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  7. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  9. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  10. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 11

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  5. 2016OpenAI Gymcited by 1,684
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  7. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  8. 2018Behavioral Cloning from Observationcited by 482
  9. 2018Distributed Prioritized Experience Replaycited by 260
  10. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 73
  11. 2020An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replaycited by 41

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles
  2. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble
  3. 2025Pretraining a Shared Q-Network for Data-Efficient Offline Reinforcement Learning
  4. 2019An Optimistic Perspective on Offline Reinforcement Learning
  5. 2023Projected Off-Policy Q-Learning (POP-QL) for Stabilizing Offline Reinforcement Learning
  6. 2024Ensemble Successor Representations for Task Generalization in Offline-to-Online Reinforcement Learning
  7. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning
  8. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning