mlsanityData to 14 Oct 2026

DeepAveragers: Offline Reinforcement Learning by Solving Derived Non-Parametric MDPs

arXiv 2010.08891 ↗First posted 18 Oct 2020cs.LG cs.AI stat.MLcited by 16

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We study an approach to offline reinforcement learning (RL) based on optimally solving finitely-represented MDPs derived from a static dataset of experience. This approach can be applied on top of any learned representation and has the potential to easily support multiple solution objectives as well as zero-shot adjustment to changing environments and goals. Our main contribution is to introduce the Deep Averagers with Costs MDP (DAC-MDP) and to investigate its solutions for offline RL. DAC-MDPs are a non-parametric model that can leverage deep representations and account for limited data by introducing costs for exploiting under-represented parts of the model. In theory, we show conditions that allow for lower-bounding the performance of DAC-MDP solutions. We also investigate the empirical behavior in a number of environments, including those with image-based observations. Overall, the experiments demonstrate that the framework can work in practice and scale to large complex offline RL problems.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 23 of its 53 references that are in the map. First posted on arXiv 18 Oct 2020.

Builds on

23 of 53 references.

Closest ancestors · 8

Same direction, newest first

  1. 2020Critic Regularized Regression26 Jun
  2. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning24 Jun
  3. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  4. 2020MOPO: Model-based Offline Policy Optimization27 May
  5. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  7. 2019Behavior Regularized Offline Reinforcement Learning26 Nov
  8. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 15

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  3. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  4. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 233
  5. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  6. 2017Neural Episodic Controlcited by 151
  7. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  8. 2016Model-Free Episodic Controlcited by 102
  9. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  10. 2020Plannable Approximations to MDP Homomorphisms: Equivariance under Actionscited by 61
  11. 2018Learning Plannable Representations with Causal InfoGANcited by 57
  12. 2018Episodic Memory Deep Q-Networkscited by 42
3 more
  1. 2020Plan2Vec: Unsupervised Representation Learning by Latent Planscited by 21
  2. 2018Efficient Model-Based Deep Reinforcement Learning with Variational State Tabulationcited by 20
  3. 2017Deep Episodic Value Iteration for Model-based Meta-Reinforcement Learningcited by 1

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 7; 2023: 2; 2024: 4; 2025: 1.

16 papers.

Sort:

By area:

  1. 2021Online and Offline Reinforcement Learning by Planning with a Learned Modelcited by 90
  2. 2022Large-Scale Retrieval for Reinforcement Learningcited by 24
  3. 2024Point Cloud Models Improve Visual Robustness in Robotic Learnerscited by 14
  4. 2022Value Memory Graph: A Graph-Structured World Model for Offline Reinforcement Learningcited by 12
  5. 2022BATS: Best Action Trajectory Stitchingcited by 10
  6. 2022Comparing Model-free and Model-based Algorithms for Offline Reinforcement Learningcited by 9
  7. 2025Generating Physically Realistic and Directable Human Motions from Multi-Modal Inputscited by 6
  8. 2022Offline Reinforcement Learning for Road Traffic Controlcited by 5
  9. 2021Cogment: Open Source Framework For Distributed Multi-actor Training, Deployment & Operationscited by 3
  10. 2023Provably Efficient Offline Reinforcement Learning with Perturbed Data Sourcescited by 3
  11. 2024Formal Ethical Obligations in Reinforcement Learning Agents: Verification and Policy Updatescited by 2
  12. 2024Efficient Exploration and Discriminative World Model Learning with an Object-Centric Abstractioncited by 2
4 more
  1. 2024Decision-Point Guided Safe Policy Improvementcited by 2
  2. 2023Contextual Pre-planning on Reward Machine Abstractions for Enhanced Transfer in Deep Reinforcement Learningcited by 1
  3. 2022Offline Policy Comparison with Confidence: Benchmarks and Baselinescited by 0
  4. 2022DCE: Offline Reinforcement Learning With Double Conservative Estimatescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  2. 2021A Minimalist Approach to Offline Reinforcement Learning
  3. 2025Offline Reinforcement Learning with Discrete Diffusion Skills
  4. 2023Offline Primal-Dual Reinforcement Learning for Linear MDPs
  5. 2024Offline RL via Feature-Occupancy Gradient Ascent
  6. 2025A Clean Slate for Offline Reinforcement Learning
  7. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  8. 2023Efficient Diffusion Policies for Offline Reinforcement Learning