mlsanityData to 7 Oct 2026

PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learning

arXiv 2106.04152 ↗First posted 8 Jun 2021cs.LGcited by 41

Not on the map. Why?

Abstract (arXiv)

Learning good feature representations is important for deep reinforcement learning (RL). However, with limited experience, RL often suffers from data inefficiency for training. For un-experienced or less-experienced trajectories (i.e., state-action sequences), the lack of data limits the use of them for better feature learning. In this work, we propose a novel method, dubbed PlayVirtual, which augments cycle-consistent virtual trajectories to enhance the data efficiency for RL feature representation learning. Specifically, PlayVirtual predicts future states in the latent space based on the current state and action by a dynamics model and then predicts the previous states by a backward dynamics model, which forms a trajectory cycle. Based on this, we augment the actions to generate a large amount of virtual state-action trajectories. Being free of groudtruth state supervision, we enforce a trajectory to meet the cycle consistency constraint, which can significantly enhance the data efficiency. We validate the effectiveness of our designs on the Atari and DeepMind Control Suite benchmarks. Our method achieves the state-of-the-art performance on both benchmarks.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 55 of its 70 references that are in the map. First posted on arXiv 8 Jun 2021.

Builds on

55 of 70 references.

References · 55

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  8. 2016Layer Normalizationcited by 5,579
  9. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  10. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  11. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  12. 2018World Modelscited by 1,444
38 more
  1. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  2. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  3. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  4. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  5. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,054
  6. 2018DeepMind Control Suitecited by 935
  7. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  8. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  9. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  10. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  11. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  12. 2019Model-Based Reinforcement Learning for Ataricited by 495
  13. 2020Reinforcement Learning with Augmented Datacited by 452
  14. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  15. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  16. 2017DualGAN: Unsupervised Dual Learning for Image-to-Image Translationcited by 310
  17. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  18. 2019Reinforcement Learning in Healthcare: A Surveycited by 308
  19. 2017Learning to Discover Cross-Domain Relations with Generative Adversarial Networkscited by 278
  20. 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
  21. 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269
  22. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  23. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  24. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  25. 2019Contrastive Learning of Structured World Modelscited by 206
  26. 2019Temporal Cycle-Consistency Learningcited by 151
  27. 2018Deep Co-Training for Semi-Supervised Image Recognitioncited by 145
  28. 2019Unsupervised State Representation Learning in Ataricited by 144
  29. 2019When to use parametric models in reinforcement learning?cited by 118
  30. 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 101
  31. 2016Learning Dense Correspondence via 3D-guided Cycle Consistencycited by 91
  32. 2020Deep Reinforcement and InfoMax Learningcited by 79
  33. 2020The Conditional Entropy Bottleneckcited by 75
  34. 2016Loss is its own Reward: Self-Supervision for Reinforcement Learningcited by 72
  35. 2020Predictive Information Accelerates Learning in RLcited by 56
  36. 2020Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistencycited by 43
  37. 2021Return-Based Contrastive Representation Learning for Reinforcement Learningcited by 36
  38. 2020Cycle-Contrast for Self-Supervised Video Representation Learningcited by 34

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 14; 2023: 9; 2024: 6; 2025: 5; 2026: 4.

41 papers.

Sort:

By area:

  1. 2022Masked World Models for Visual Controlcited by 169
  2. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  3. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 58
  4. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  5. 2025Towards General-Purpose Model-Free Reinforcement Learningcited by 40
  6. 2022Mask-based Latent Reconstruction for Reinforcement Learningcited by 39
  7. 2023Crossway Diffusion: Improving Diffusion-based Visuomotor Policy via Self-supervised Learningcited by 37
  8. 2023Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stagescited by 27
  9. 2022Does Self-supervised Learning Really Improve Reinforcement Learning from Pixels?cited by 24
  10. 2022A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learningcited by 23
  11. 2022Planning for Sample Efficient Imitation Learningcited by 20
  12. 2023Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RLcited by 18
29 more
  1. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  2. 2022An Empirical Investigation of Representation Learning for Imitationcited by 15
  3. 2022Value-Consistent Representation Learning for Data-Efficient Reinforcement Learningcited by 15
  4. 2022Class-Level Confidence Based 3D Semi-Supervised Learningcited by 13
  5. 2023Robust Representation Learning by Clustering with Bisimulation Metrics for Visual Reinforcement Learning with Distractionscited by 13
  6. 2023Learning Better with Less: Effective Augmentation for Sample-Efficient Visual Reinforcement Learningcited by 8
  7. 2021Self-Consistent Models and Valuescited by 7
  8. 2022Robust Imitation of a Few Demonstrations with a Backwards Modelcited by 7
  9. 2021Model-Value Inconsistency as a Signal for Epistemic Uncertaintycited by 6
  10. 2023Successor-Predecessor Intrinsic Explorationcited by 6
  11. 2023MA2CL:Masked Attentive Contrastive Learning for Multi-Agent Reinforcement Learningcited by 6
  12. 2021Learning State Representations via Retracing in Reinforcement Learningcited by 5
  13. 2024Investigating Pre-Training Objectives for Generalization in Vision-Based Reinforcement Learningcited by 4
  14. 2022Accelerating Representation Learning with View-Consistent Dynamics in Data-Efficient Reinforcement Learningcited by 3
  15. 2024Learning Future Representation with Synthetic Observations for Sample-efficient Reinforcement Learningcited by 3
  16. 2024MOOSS: Mask-Enhanced Temporal Contrastive Learning for Smooth State Evolution in Visual Reinforcement Learningcited by 3
  17. 2025Revisiting Bisimulation Metric for Robust Representations in Reinforcement Learningcited by 3
  18. 2025PvP: Data-Efficient Humanoid Robot Learning with Proprioceptive-Privileged Contrastive Representationscited by 2
  19. 2026ResWM: Residual-Action World Model for Visual RLcited by 2
  20. 2024DyConfidMatch: Dynamic Thresholding and Re-sampling for 3D Semi-supervised Learningcited by 1
  21. 2026COVR:Collaborative Optimization of VLMs and RL Agent for Visual-Based Controlcited by 1
  22. 2022Unified State Representation Learning under Data Augmentationcited by 0
  23. 2022Using Forwards-Backwards Models to Approximate MDP Homomorphismscited by 0
  24. 2023Towards Control-Centric Representations in Reinforcement Learning from Imagescited by 0
  25. 2024ProSpec RL: Plan Ahead, then Executecited by 0
  26. 2024Uncovering RL Integration in SSL Loss: Objective-Specific Implications for Data-Efficient RLcited by 0
  27. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategiescited by 0
  28. 2026Mask-based Predictive Representations for Reinforcement Learningcited by 0
  29. 2026V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Controlcited by 0