mlsanityData to 14 Oct 2026

DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning

arXiv 2203.06662 ↗First posted 13 Mar 2022cs.LGcited by 41

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning algorithms promise to be applicable in settings where a fixed dataset is available and no new experience can be acquired. However, such formulation is inevitably offline-data-hungry and, in practice, collecting a large offline dataset for one specific task over one specific environment is also costly and laborious. In this paper, we thus 1) formulate the offline dynamics adaptation by using (source) offline data collected from another dynamics to relax the requirement for the extensive (target) offline data, 2) characterize the dynamics shift problem in which prior offline methods do not scale well, and 3) derive a simple dynamics-aware reward augmentation (DARA) framework from both model-free and model-based offline settings. Specifically, DARA emphasizes learning from those source transition pairs that are adaptive for the target environment and mitigates the offline dynamics shift by characterizing state-action-next-state pairs instead of the typical state-action distribution sketched by prior offline RL methods. The experimental evaluation demonstrates that DARA, by augmenting rewards in the source offline dataset, can acquire an adaptive policy for the target environment and yet significantly reduce the requirement of target offline data. With only modest amounts of target offline data, our performance consistently outperforms the prior offline RL methods in both simulated and real-world tasks.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 43 of its 52 references that are in the map. First posted on arXiv 13 Mar 2022.

Builds on

43 of 52 references.

Closest ancestors · 13

Same direction, newest first

  1. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL16 Jun
  2. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  3. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  4. 2021NeoRL: A Near Real-World Benchmark for Offline Reinforcement Learning1 Feb
  5. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  6. 2020Critic Regularized Regression26 Jun
  7. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  8. 2020MOPO: Model-based Offline Policy Optimization27 May
  9. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  12. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
1 more
  1. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 30

  1. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  3. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  6. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  8. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 604
  9. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  10. 2018Maximum a Posteriori Policy Optimisationcited by 328
  11. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  12. 2019A Theory of Regularized Markov Decision Processescited by 286
18 more
  1. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  2. 2018Breaking the Curse of Horizon: Infinite-Horizon Off-Policy Estimationcited by 227
  3. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  4. 2019Minimax Weight and Q-Function Learning for Off-Policy Evaluationcited by 150
  5. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  6. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  7. 2020Reinforcement Learning via Fenchel-Rockafellar Dualitycited by 100
  8. 2020Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifierscited by 97
  9. 2020Offline Meta-Reinforcement Learning with Advantage Weightingcited by 93
  10. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 91
  11. 2021Constraints Penalized Q-learning for Safe Offline Reinforcement Learningcited by 91
  12. 2020Off-Policy Evaluation via the Regularized Lagrangiancited by 86
  13. 2020One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RLcited by 79
  14. 2019Policies Modulating Trajectory Generatorscited by 49
  15. 2021Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environmentcited by 42
  16. 2019If MaxEnt RL is the Answer, What is the Question?cited by 27
  17. 2021Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learningcited by 15
  18. 2021Discriminator Augmented Model-Based Reinforcement Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 11; 2024: 12; 2025: 10; 2026: 6.

41 papers.

Sort:

By area:

  1. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learningcited by 60
  2. 2023Behavior Proximal Policy Optimizationcited by 43
  3. 2023ChiPFormer: Transferable Chip Placement via Offline Decision Transformercited by 42
  4. 2023Beyond Reward: Offline Preference-guided Policy Optimizationcited by 35
  5. 2023Cross-Domain Policy Adaptation via Value-Guided Data Filteringcited by 31
  6. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learningcited by 27
  7. 2024ODRL: A Benchmark for Off-Dynamics Reinforcement Learningcited by 25
  8. 2023State Regularized Policy Optimization on Data with Dynamics Shiftcited by 23
  9. 2023H2O+: An Improved Framework for Hybrid Offline-and-Online RL with Dynamics Gapscited by 17
  10. 2024Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitationcited by 17
  11. 2025Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuningcited by 17
  12. 2024A Comprehensive Survey of Cross-Domain Policy Transfer for Embodied Agentscited by 16
29 more
  1. 2023CEIL: Generalized Contextual Imitation Learningcited by 13
  2. 2024Return Augmented Decision Transformer for Off-Dynamics Reinforcement Learningcited by 13
  3. 2023CLUE: Calibrated Latent Guidance for Offline Reinforcement Learningcited by 11
  4. 2024Scalable and Effective Arithmetic Tree Generation for Adder and Multiplier Designscited by 9
  5. 2024xTED: Cross-Domain Adaptation via Diffusion-Based Trajectory Editingcited by 9
  6. 2025Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Datacited by 9
  7. 2025MOBODY: Model Based Off-Dynamics Offline Reinforcement Learningcited by 9
  8. 2023Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimizationcited by 8
  9. 2024Hybrid Transfer Reinforcement Learning: Provable Sample Efficiency from Shifted-Dynamics Datacited by 8
  10. 2024Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learningcited by 7
  11. 2025DmC: Nearest Neighbor Guidance Diffusion Model for Offline Cross-domain Reinforcement Learningcited by 7
  12. 2025Dual-Robust Cross-Domain Offline Reinforcement Learning Against Dynamics Shiftscited by 6
  13. 2024Offline Reinforcement Learning from Datasets with Structured Non-Stationaritycited by 5
  14. 2024Provable Domain Adaptation for Offline Reinforcement Learning with Limited Samplescited by 5
  15. 2025Attention-Based Reward Shaping for Sparse and Delayed Rewardscited by 5
  16. 2023OER: Offline Experience Replay for Continual Offline Reinforcement Learningcited by 4
  17. 2024Offline Reinforcement Learning with Domain-Unlabeled Datacited by 3
  18. 2025Hybrid Cross-domain Robust Reinforcement Learningcited by 3
  19. 2025A Multi-Fidelity Control Variate Approach for Policy Gradient Estimationcited by 2
  20. 2025Clustering-Based Weight Orthogonalization for Stabilizing Deep Reinforcement Learningcited by 2
  21. 2022KSG: Knowledge and Skill Graphcited by 1
  22. 2026Localized Dynamics-Aware Domain Adaption for Off-Dynamics Offline Reinforcement Learningcited by 1
  23. 2024Off-dynamics Conditional Diffusion Plannerscited by 0
  24. 2025Target-Aligned Fusion for Decision-Sequence Learning under Dynamics Shiftcited by 0
  25. 2026Cross-Domain Policy Optimization via Bellman Consistency and Hybrid Criticscited by 0
  26. 2026Bridging Domain Gaps with Target-Aligned Generation for Offline Reinforcement Learningcited by 0
  27. 2026Target-Aligned Bellman Backup for Cross-domain Offline Reinforcement Learningcited by 0
  28. 2026Cross-Domain Energy-Guided Diffusion Generation for Off-Dynamics Reinforcement Learningcited by 0
  29. 2026Multi-Fidelity Policy Gradients Stabilize Data-Scarce Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning
  2. 2025Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning
  3. 2024Augmenting Offline Reinforcement Learning with State-only Interactions
  4. 2026Trajectory-Level Data Augmentation for Offline Reinforcement Learning
  5. 2025A Clean Slate for Offline Reinforcement Learning
  6. 2022Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief
  7. 2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models
  8. 2023Exploiting Generalization in Offline Reinforcement Learning via Unseen State Augmentations