mlsanityData to 14 Oct 2026

Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents

arXiv 2510.13704 ↗First posted 15 Oct 2025cs.LG cs.AI cs.ROcited by 12

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Recent works have proposed accelerating the wall-clock training time of actor-critic methods via the use of large-scale environment parallelization; unfortunately, these can sometimes still require large number of environment interactions to achieve a desired level of performance. Noting that well-structured representations can improve the generalization and sample efficiency of deep reinforcement learning (RL) agents, we propose the use of simplicial embeddings: lightweight representation layers that constrain embeddings to simplicial structures. This geometric inductive bias results in sparse and discrete features that stabilize critic bootstrapping and strengthen policy gradients. When applied to FastTD3, FastSAC, and PPO, simplicial embeddings consistently improve sample efficiency and final performance across a variety of continuous- and discrete-control environments, without any loss in runtime speed.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 106 of its 132 references that are in the map. First posted on arXiv 15 Oct 2025.

Builds on

106 of 132 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learning8 Mar
  2. 2024Non-Stationary Learning of Neural Networks with Automatic Soft Parameter Reset6 Nov
  3. 2024SAPG: Split and Aggregate Policy Gradients29 Jul
  4. 2024Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning1 Mar

Other references · 102

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2017Neural Discrete Representation Learningcited by 4,712
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  7. 2015Continuous control with deep reinforcement learningcited by 3,407
  8. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  9. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  10. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  11. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  12. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
38 more
  1. 2023Mastering Diverse Domains through World Modelscited by 1,253
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  3. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
  4. 2020Mastering Atari with Discrete World Modelscited by 1,007
  5. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  6. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  7. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  8. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  9. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  10. 2019Solving Rubik's Cube with a Robot Handcited by 734
  11. 2019Green AIcited by 627
  12. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  13. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 512
  14. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  15. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  16. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  17. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 369
  18. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  19. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 300
  20. 2023Orbit: A Unified Simulation Framework for Interactive Robot Learning Environmentscited by 295
  21. 2020Towards the Systematic Reporting of the Energy and Carbon Footprints of Machine Learningcited by 249
  22. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  23. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  24. 2021Mastering Atari Games with Limited Datacited by 237
  25. 2024Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learningcited by 189
  26. 2025Flow Q-Learningcited by 184
  27. 2020Revisiting Fundamentals of Experience Replaycited by 175
  28. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 174
  29. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  30. 2023Understanding plasticity in neural networkscited by 174
  31. 2022Rapid Locomotion via Reinforcement Learningcited by 168
  32. 2023Robot Parkour Learningcited by 149
  33. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  34. 2019Unsupervised State Representation Learning in Ataricited by 144
  35. 2020Implicit Under-Parameterization Inhibits Data-Efficient Deep Reinforcement Learningcited by 134
  36. 2018State Representation Learning for Control: An Overviewcited by 127
  37. 2022Understanding and Preventing Capacity Loss in Reinforcement Learningcited by 125
  38. 2023Loss of Plasticity in Continual Deep Reinforcement Learningcited by 121

50 most cited of 102 All 102 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 11.

12 papers.

  1. 2025Learning Sim-to-Real Humanoid Locomotion in 15 Minutescited by 16
  2. 2026The Surprising Difficulty of Search in Model-Based Reinforcement Learningcited by 6
  3. 2026Stable Deep Reinforcement Learning via Isotropic Gaussian Representationscited by 5
  4. 2026FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Controlcited by 5
  5. 2026Squint: Fast Visual Reinforcement Learning for Sim-to-Real Roboticscited by 2
  6. 2026FastDSAC: Unlocking the Potential of Maximum Entropy RL in High-Dimensional Humanoid Controlcited by 1
  7. 2026Debiased Model-based Representations for Sample-efficient Continuous Controlcited by 1
  8. 2026Temporal Consistency Improves Generalization in Contextual Offline Meta Reinforcement Learningcited by 0
  9. 2026Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitionscited by 0
  10. 2026Representation Learning Enables Scalable Multitask Deep Reinforcement Learningcited by 0
  11. 2026Beyond One-Size-Fits-All: Diagnosis-Driven Online Reinforcement Learning with Offline Priorscited by 0
  12. 2026V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Controlcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learning
  2. 2026Simple Actors and Deep Critics for Scalable Reinforcement Learning
  3. 2026Scalable Neighborhood-Based Multi-Agent Actor-Critic
  4. 2026XQCfD: Accelerating Fast Actor-Critic Algorithms with Prior Data and Prior Policies
  5. 2025Distributions as Actions: A Unified Framework for Diverse Action Spaces
  6. 2025Gradient Coupling: The Hidden Barrier to Generalization in Agentic Reinforcement Learning
  7. 2025Understanding and Improving Hyperbolic Deep Reinforcement Learning
  8. 2026RN-D: Discretized Categorical Actors for On-Policy Reinforcement Learning