mlsanityData to 7 Oct 2026

For SALE: State-Action Representation Learning for Deep Reinforcement Learning

arXiv 2306.02451 ↗First posted 4 Jun 2023cs.LG cs.AI stat.MLcited by 73

Not on the map. Why?

Abstract (arXiv)

In the field of reinforcement learning (RL), representation learning is a proven tool for complex image-based tasks, but is often overlooked for environments with low-level states, such as physical control problems. This paper introduces SALE, a novel approach for learning embeddings that model the nuanced interaction between state and action, enabling effective representation learning from low-level states. We extensively study the design space of these embeddings and highlight important design considerations. We integrate SALE and an adaptation of checkpoints for RL into TD3 to form the TD7 algorithm, which significantly outperforms existing continuous control algorithms. On OpenAI gym benchmark tasks, TD7 has an average performance gain of 276.7% and 50.7% over TD3 at 300k and 5M time steps, respectively, and works in both the online and offline settings.

Key facts. Cited by 73 papers in the map (data through 7 Oct 2026). Builds on 61 of its 80 references that are in the map. First posted on arXiv 4 Jun 2023.

Builds on

61 of 80 references.

References · 61

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2017Proximal Policy Optimization Algorithmscited by 14,441
  6. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  9. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  10. 2015Continuous control with deep reinforcement learningcited by 3,279
  11. 2015Trust Region Policy Optimizationcited by 2,503
  12. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
38 more
  1. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  2. 2016OpenAI Gymcited by 1,684
  3. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  6. 2023Mastering Diverse Domains through World Modelscited by 1,193
  7. 2015Prioritized Experience Replaycited by 1,070
  8. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,061
  9. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  10. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  11. 2017Deep Reinforcement Learning that Matterscited by 727
  12. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  13. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
  14. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  15. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  16. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  17. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  18. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  19. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  20. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
  21. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  22. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  23. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
  24. 2016Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Datacited by 154
  25. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 154
  26. 2015Deep Spatial Autoencoders for Visuomotor Learningcited by 149
  27. 2019Unsupervised State Representation Learning in Ataricited by 144
  28. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  29. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  30. 2020MDP Homomorphic Networks: Group Symmetries in Reinforcement Learningcited by 133
  31. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 124
  32. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 113
  33. 2021Benchmarks for Deep Off-Policy Evaluationcited by 89
  34. 2018Evolution-Guided Policy Gradient in Reinforcement Learningcited by 78
  35. 2018Simple random search provides a competitive approach to reinforcement learningcited by 75
  36. 2019Learning Action Representations for Reinforcement Learningcited by 75
  37. 2020Plannable Approximations to MDP Homomorphisms: Equivariance under Actionscited by 61
  38. 2018CEM-RL: Combining evolutionary and gradient-based methods for policy searchcited by 53

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 20; 2025: 25; 2026: 25.

73 papers.

Sort:

By area:

  1. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learningcited by 70
  2. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 52
  3. 2025Towards General-Purpose Model-Free Reinforcement Learningcited by 40
  4. 2025Hyperspherical Normalization for Scalable Deep Reinforcement Learningcited by 35
  5. 2024Cross-Domain Policy Adaptation by Capturing Representation Mismatchcited by 24
  6. 2025Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learnerscited by 23
  7. 2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
  8. 2024Neuroplastic Expansion in Deep Reinforcement Learningcited by 17
  9. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  10. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
  11. 2024Diffusion Spectral Representation for Reinforcement Learningcited by 15
  12. 2025Action Tokenizer Matters in In-Context Imitation Learningcited by 12
38 more
  1. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
  2. 2025Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learningcited by 11
  3. 2024Efficient Recurrent Off-Policy RL Requires a Context-Encoder-Specific Learning Ratecited by 10
  4. 2024Non-Adversarial Inverse Reinforcement Learning via Successor Feature Matchingcited by 10
  5. 2024iQRL -- Implicitly Quantized Representations for Sample-efficient Reinforcement Learningcited by 9
  6. 2025MOBODY: Model Based Off-Dynamics Offline Reinforcement Learningcited by 8
  7. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
  8. 2024Value Improved Actor Critic Algorithmscited by 6
  9. 2025Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learningcited by 5
  10. 2026The Surprising Difficulty of Search in Model-Based Reinforcement Learningcited by 5
  11. 2026FLAC: Maximum Entropy RL via Kinetic Energy Regularized Bridge Matchingcited by 5
  12. 2026Stable Deep Reinforcement Learning via Isotropic Gaussian Representationscited by 5
  13. 2026FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Controlcited by 5
  14. 2024Simple Ingredients for Offline Reinforcement Learningcited by 4
  15. 2026Improving Policy Exploitation in Online Reinforcement Learning with Instant Retrospect Actioncited by 4
  16. 2023Generalizable Imitation Learning Through Pre-Trained Representationscited by 3
  17. 2024Offline Reinforcement Learning with Domain-Unlabeled Datacited by 3
  18. 2024Feasibility Consistent Representation Learning for Safe Reinforcement Learningcited by 3
  19. 2025SR-Reward: Taking The Path More Traveledcited by 3
  20. 2025HypeRL: Hypernetwork-Based Reinforcement Learning for Control of Parametrized Dynamical Systemscited by 3
  21. 2025Spectral Representation-based Reinforcement Learningcited by 3
  22. 2024Double Actor-Critic with TD Error-Driven Regularization in Reinforcement Learningcited by 2
  23. 2025Optimizing Likelihoods via Mutual Information: Bridging Simulation-Based Inference and Bayesian Optimal Experimental Designcited by 2
  24. 2025Sparse-Reg: Improving Sample Complexity in Offline Reinforcement Learning using Sparsitycited by 2
  25. 2025Human-Like Goalkeeping in a Realistic Football Simulation: a Sample-Efficient Reinforcement Learning Approachcited by 2
  26. 2025Mind Your Entropy: From Maximum Entropy to Trajectory Entropy-Constrained RLcited by 2
  27. 2026Temporal Difference Learning with Constrained Initial Representationscited by 2
  28. 2023Efficient Sparse-Reward Goal-Conditioned Reinforcement Learning with a High Replay Ratio and Regularizationcited by 1
  29. 2025Universal Value-Function Uncertaintiescited by 1
  30. 2025Stackelberg Coupling of Online Representation Learning and Reinforcement Learningcited by 1
  31. 2025Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learningcited by 1
  32. 2026SIA: Symbolic Interpretability for Anticipatory Deep Reinforcement Learning in Network Controlcited by 1
  33. 2026MSPR: Multi-scale Predictive Representations for Goal-conditioned Reinforcement Learningcited by 1
  34. 2026Debiased Model-based Representations for Sample-efficient Continuous Controlcited by 1
  35. 2026Spectral Souping: A Unified Framework for Online Preference Alignmentcited by 1
  36. 2026Compositional Transduction with Latent Analogies for Offline Goal-Conditioned Reinforcement Learningcited by 1
  37. 2026SMC-ES: Automated synthesis of formally verified control policiescited by 1
  38. 2024SQT -- std Q-targetcited by 0

50 most cited of 73 All 73 (JSON)