mlsanityData to 7 Oct 2026

S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning

arXiv 2103.06326 ↗First posted 10 Mar 2021cs.LGcited by 85

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning proposes to learn policies from large collected datasets without interacting with the physical environment. These algorithms have made it possible to learn useful skills from data that can then be deployed in the environment in real-world settings where interactions may be costly or dangerous, such as autonomous driving or factories. However, current algorithms overfit to the dataset they are trained on and exhibit poor out-of-distribution generalization to the environment when deployed. In this paper, we study the effectiveness of performing data augmentations on the state space, and study 7 different augmentation schemes and how they behave with existing offline RL algorithms. We then combine the best data performing augmentation scheme with a state-of-the-art Q-learning technique, and improve the function approximation of the Q-networks by smoothening out the learned state-action space. We experimentally show that using this Surprisingly Simple Self-Supervision technique in RL (S4RL), we significantly improve over the current state-of-the-art algorithms on offline robot learning environments such as MetaWorld [1] and RoboSuite [2,3], and benchmark datasets such as D4RL [4].

Key facts. Cited by 85 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 48 of its 66 references that are in the map. First posted on arXiv 10 Mar 2021.

Builds on

48 of 66 references.

Closest ancestors · 8

Same direction, newest first

  1. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning26 Oct
  2. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning24 Jun
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020MOPO: Model-based Offline Policy Optimization27 May
  5. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  8. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 40

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  3. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  7. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  8. 2016Context Encoders: Feature Learning by Inpaintingcited by 1,424
  9. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
  10. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  11. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  12. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
28 more
  1. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  3. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  4. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  5. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  6. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  7. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  8. 2018Behavioral Cloning from Observationcited by 482
  9. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  10. 2020Reinforcement Learning with Augmented Datacited by 452
  11. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  12. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  13. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  14. 2019Stabilizing Transformers for Reinforcement Learningcited by 262
  15. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  16. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 202
  17. 2017DART: Noise Injection for Robust Imitation Learningcited by 187
  18. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  19. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  20. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  21. 2019Self-Supervised Correspondence in Visuomotor Policy Learningcited by 106
  22. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  23. 2020Counterfactual Data Augmentation using Locally Factored Dynamicscited by 93
  24. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 85
  25. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  26. 2020Experience Replay with Likelihood-free Importance Weightscited by 42
  27. 2020D2RL: Deep Dense Architectures in Reinforcement Learningcited by 40
  28. 2019Scaling data-driven robotics with reward sketching and batch reinforcement learningcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 14; 2023: 23; 2024: 21; 2025: 13; 2026: 7.

85 papers.

Sort:

By area:

  1. 2021Implicit Behavioral Cloningcited by 442
  2. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  3. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learningcited by 114
  4. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 105
  5. 2024DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learningcited by 104
  6. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  7. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
  8. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
  9. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothingcited by 83
  10. 2023Elastic Decision Transformercited by 66
  11. 2023Constrained Decision Transformer for Offline Safe Reinforcement Learningcited by 65
  12. 2023Datasets and Benchmarks for Offline Safe Reinforcement Learningcited by 58
38 more
  1. 2021Consistency Regularization for Variational Auto-Encoderscited by 50
  2. 2022Hierarchical Planning Through Goal-Conditioned Offline Reinforcement Learningcited by 48
  3. 2024SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deploymentcited by 48
  4. 2022Bootstrapped Transformer for Offline Reinforcement Learningcited by 37
  5. 2021Medical Dead-ends and Learning to Identify High-risk States and Treatmentscited by 28
  6. 2021Koopman Q-learning: Offline Reinforcement Learning via Symmetries of Dynamicscited by 27
  7. 2022Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?cited by 24
  8. 2022Continuous MDP Homomorphisms and Homomorphic Policy Gradientcited by 24
  9. 2024GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learningcited by 22
  10. 2022Semi-Markov Offline Reinforcement Learning for Healthcarecited by 18
  11. 2023Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RLcited by 18
  12. 2021Improving Zero-shot Generalization in Offline Reinforcement Learning using Generalized Similarity Functionscited by 17
  13. 2023LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speechcited by 17
  14. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learningcited by 14
  15. 2025Generative Models in Decision Making: A Surveycited by 14
  16. 2022Towards Informed Design and Validation Assistance in Computer Games Using Imitation Learningcited by 13
  17. 2023Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learningcited by 13
  18. 2022Latent-Variable Advantage-Weighted Policy Optimization for Offline RLcited by 12
  19. 2023A Survey of Demonstration Learningcited by 12
  20. 2023Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothnesscited by 12
  21. 2024RoCoDA: Counterfactual Data Augmentation for Data-Efficient Robot Learning from Demonstrationscited by 12
  22. 2022A Behavior Regularized Implicit Policy for Offline Reinforcement Learningcited by 11
  23. 2025State-Covering Trajectory Stitching for Diffusion Plannerscited by 11
  24. 2023Guided Data Augmentation for Offline Reinforcement Learning and Imitation Learningcited by 10
  25. 2024OASIS: Conditional Distribution Shaping for Offline Safe Reinforcement Learningcited by 10
  26. 2024xTED: Cross-Domain Adaptation via Diffusion-Based Trajectory Editingcited by 9
  27. 2023Mixed Traffic Control and Coordination from Pixelscited by 8
  28. 2024Offline Imitation Learning with Model-based Reverse Augmentationcited by 8
  29. 2024Align Your Intents: Offline Imitation Learning via Optimal Transportcited by 7
  30. 2023Learning non-Markovian Decision-Making from State-only Sequencescited by 6
  31. 2023Understanding when Dynamics-Invariant Data Augmentations Benefit Model-Free Reinforcement Learning Updatescited by 6
  32. 2024PlanDQ: Hierarchical Plan Orchestration via D-Conductor and Q-Performercited by 6
  33. 2024Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformercited by 6
  34. 2023Identifying Expert Behavior in Offline Training Datasets Improves Behavioral Cloning of Robotic Manipulation Policiescited by 5
  35. 2024Efficient Policy Adaptation with Contrastive Prompt Ensemble for Embodied Agentscited by 5
  36. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learningcited by 5
  37. 2025Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learningcited by 5
  38. 2023Improving Generalization in Game Agents with Data Augmentation in Imitation Learningcited by 4

50 most cited of 85 All 85 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Pretraining a Shared Q-Network for Data-Efficient Offline Reinforcement Learning
  2. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning
  3. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning
  4. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  5. 2023Small Dataset, Big Gains: Enhancing Reinforcement Learning by Offline Pre-Training with Model Based Augmentation
  6. 2021A Minimalist Approach to Offline Reinforcement Learning
  7. 2024D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning
  8. 2022Real World Offline Reinforcement Learning with Realistic Data Source