mlsanityData to 7 Oct 2026

Look Beneath the Surface: Exploiting Fundamental Symmetry for Sample-Efficient Offline RL

arXiv 2306.04220 ↗First posted 7 Jun 2023cs.LG cs.AIcited by 18

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) offers an appealing approach to real-world tasks by learning policies from pre-collected datasets without interacting with the environment. However, the performance of existing offline RL algorithms heavily depends on the scale and state-action space coverage of datasets. Real-world data collection is often expensive and uncontrollable, leading to small and narrowly covered datasets and posing significant challenges for practical deployments of offline RL. In this paper, we provide a new insight that leveraging the fundamental symmetry of system dynamics can substantially enhance offline RL performance under small datasets. Specifically, we propose a Time-reversal symmetry (T-symmetry) enforced Dynamics Model (TDM), which establishes consistency between a pair of forward and reverse latent dynamics. TDM provides both well-behaved representations for small datasets and a new reliability measure for OOD samples based on compliance with the T-symmetry. These can be readily used to construct a new offline RL algorithm (TSRL) with less conservative policy constraints and a reliable latent space data augmentation procedure. Based on extensive experiments, we find TSRL achieves great performance on small benchmark datasets with as few as 1% of the original samples, which significantly outperforms the recent offline RL algorithms in terms of data efficiency and generalizability.Code is available at: https://github.com/pcheng2/TSRL

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 45 of its 65 references that are in the map. First posted on arXiv 7 Jun 2023.

Builds on

45 of 65 references.

Closest ancestors · 25

Same direction, newest first

  1. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  2. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning27 Jun
  3. 2022Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imagination16 Jun
  4. 2022When Data Geometry Meets Deep Function: Generalizing Offline Reinforcement Learning23 May
  5. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  6. 2021Koopman Q-learning: Offline Reinforcement Learning via Symmetries of Dynamics2 Nov
  7. 2021Offline Reinforcement Learning with Soft Behavior Regularization14 Oct
  8. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  9. 2021Representation Learning for Online and Offline RL in Low-rank MDPs9 Oct
  10. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  11. 2021Offline Reinforcement Learning with Reverse Model-based Imagination1 Oct
  12. 2021Offline RL Without Off-Policy Evaluation16 Jun
13 more
  1. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  2. 2021Model-Based Offline Planning with Trajectory Pruning16 May
  3. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  4. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning10 Mar
  5. 2021Representation Matters: Offline Pretraining for Sequential Decision Making11 Feb
  6. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  7. 2020Critic Regularized Regression26 Jun
  8. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  9. 2020MOPO: Model-based Offline Policy Optimization27 May
  10. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  13. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 20

  1. 2018Neural Ordinary Differential Equationscited by 3,733
  2. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  7. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  8. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  9. 2019Augmented Neural ODEscited by 396
  10. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 285
  11. 2019Batch Policy Learning under Constraintscited by 242
  12. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
8 more
  1. 2019Neural SDE: Stabilizing Neural ODE Networks with Stochastic Noisecited by 98
  2. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 90
  3. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learningcited by 58
  4. 2021DeepThermal: Combustion Optimization for Thermal Power Generating Units Using Offline Reinforcement Learningcited by 47
  5. 2019Imitation Learning from Observations by Minimizing Inverse Dynamics Disagreementcited by 46
  6. 2021PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learningcited by 41
  7. 2020Time-Reversal Symmetric ODE Networkcited by 19
  8. 2021Learning State Representations via Retracing in Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 4; 2025: 9; 2026: 3.

18 papers.

Sort:

By area:

  1. 2024Safe Offline Reinforcement Learning with Feasibility-Guided Diffusion Modelcited by 62
  2. 2023Offline Multi-Agent Reinforcement Learning with Implicit Global-to-Local Value Regularizationcited by 42
  3. 2024ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Updatecited by 21
  4. 2025Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learningcited by 15
  5. 2025Skill Expansion and Composition in Parameter Spacecited by 13
  6. 2024ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularizationcited by 8
  7. 2025Towards Robust Zero-Shot Reinforcement Learningcited by 7
  8. 2025Efficient Whole Slide Pathology VQA via Token Compressioncited by 6
  9. 2025Data Center Cooling System Optimization Using Offline Reinforcement Learningcited by 4
  10. 2025An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learningcited by 4
  11. 2024SiT: Symmetry-Invariant Transformers for Generalisation in Reinforcement Learningcited by 3
  12. 2025Sample-Efficient Behavior Cloning Using General Domain Knowledgecited by 2
6 more
  1. 2023An Investigation of Time Reversal Symmetry in Reinforcement Learningcited by 1
  2. 2026Mollified Value Learningcited by 1
  3. 2025Time Reversal Symmetry for Efficient Robotic Manipulations in Deep Reinforcement Learningcited by 0
  4. 2025An Empirical Study on the Effectiveness of Incorporating Offline RL As Online RL Subroutinescited by 0
  5. 2026Horizon Adaptive Offline Policy Learning via Value Stitchingcited by 0
  6. 2026Generalization in offline RL: The structure is more important than the amount of pessimismcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  2. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  3. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning
  4. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  5. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  6. 2025Efficient Cross-Domain Offline Reinforcement Learning with Dynamics- and Value-Aligned Data Filtering
  7. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
  8. 2020Offline Reinforcement Learning from Images with Latent Space Models