mlsanityData to 14 Oct 2026

Debiased Model-based Representations for Sample-efficient Continuous Control

arXiv 2605.11711 ↗First posted 12 May 2026cs.LG cs.AIcited by 1

On the map: Deep reinforcement learning methods → RL theory and sample complexity

Abstract (arXiv)

Model-based representations recently stand out as a promising framework that embeds latent dynamics information into the representations for downstream off-policy actor-critic learning. It implicitly combines the advantages of both model-free and model-based approaches while avoiding the training costs associated with model-based methods. Nevertheless, existing model-based representation methods can fail to capture sufficient information about relevant variables and can overfit to early experiences in the replay buffer. These incur biases in representation and actor-critic learning, leading to inferior performance. To address this, we propose Debiased model-based Representations for Q-learning, tagged DR.Q algorithm. DR.Q explicitly maximizes the mutual information between the representations of the current state-action pair and the next state besides minimizing their deviations, and samples transitions with faded prioritized experience replay. We evaluate DR.Q on numerous continuous control benchmarks with a single set of hyperparameters, and the results demonstrate that DR.Q can match or surpass recent strong baselines, sometimes outperforming them by a large margin. Our code is available at https://github.com/dmksjfl/DR.Q.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods. Builds on 94 of its 114 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

94 of 114 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents15 Oct
  2. 2025First Order Model-Based RL through Decoupled Backpropagation29 Aug

Other references · 92

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  5. 2015Continuous control with deep reinforcement learningcited by 3,407
  6. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  7. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  8. 2016OpenAI Gymcited by 1,736
  9. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  10. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  11. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  12. 2023Mastering Diverse Domains through World Modelscited by 1,253
38 more
  1. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  2. 2017Hindsight Experience Replaycited by 1,136
  3. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  4. 2018Exploration by Random Network Distillationcited by 993
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  6. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  7. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  8. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  9. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  10. 2019On Variational Bounds of Mutual Informationcited by 559
  11. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  12. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  13. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  14. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  15. 2019On Mutual Information Maximization for Representation Learningcited by 316
  16. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  17. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 274
  18. 2018Distributed Prioritized Experience Replaycited by 269
  19. 2020Decoupling Representation Learning from Reinforcement Learningcited by 269
  20. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  21. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 231
  22. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 207
  23. 2018Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansioncited by 168
  24. 2022Exploration in Deep Reinforcement Learning: A Surveycited by 166
  25. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 165
  26. 2016Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Datacited by 156
  27. 2015Deep Spatial Autoencoders for Visuomotor Learningcited by 153
  28. 2018State Representation Learning for Control: An Overviewcited by 127
  29. 2021Dropout Q-Functions for Doubly Efficient Reinforcement Learningcited by 114
  30. 2024HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulationcited by 112
  31. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 108
  32. 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 102
  33. 2019CrossQ: Batch Normalization in Deep Reinforcement Learning for Greater Sample Efficiency and Simplicitycited by 95
  34. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  35. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 75
  36. 2022BYOL-Explore: Exploration by Bootstrapped Predictioncited by 73
  37. 2024Learning and Leveraging World Models in Visual Representation Learningcited by 72
  38. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 65

50 most cited of 92 All 92 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026EfficientTDMPC: Improved MPC Objectives for Sample-Efficient Continuous Controlcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2025Actor-Free Continuous Control via Structurally Maximizable Q-Functions
  2. 2025Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Network
  3. 2025Mitigating Estimation Bias with Representation Learning in TD Error-Driven Regularization
  4. 2024AFU: Actor-Free critic Updates in off-policy RL for continuous control
  5. 2026Cost-Driven Representation Learning for Linear Quadratic Gaussian Control: Part II
  6. 2025Langevin Soft Actor-Critic: Efficient Exploration through Uncertainty-Driven Critic Learning
  7. 2024MinMaxMin Q-learning
  8. 2025SALE-Based Offline Reinforcement Learning with Ensemble Q-Networks