mlsanityData to 14 Oct 2026

Design from Policies: Conservative Test-Time Adaptation for Offline Policy Optimization

arXiv 2306.14479 ↗First posted 26 Jun 2023cs.LGcited by 8

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In this work, we decouple the iterative bi-level offline RL (value estimation and policy extraction) from the offline training phase, forming a non-iterative bi-level paradigm and avoiding the iterative error propagation over two levels. Specifically, this non-iterative paradigm allows us to conduct inner-level optimization (value estimation) in training, while performing outer-level optimization (policy extraction) in testing. Naturally, such a paradigm raises three core questions that are not fully answered by prior non-iterative offline RL counterparts like reward-conditioned policy: (q1) What information should we transfer from the inner-level to the outer-level? (q2) What should we pay attention to when exploiting the transferred information for safe/confident outer-level optimization? (q3) What are the benefits of concurrently conducting outer-level optimization during testing? Motivated by model-based optimization (MBO), we propose DROP (design from policies), which fully answers the above questions. Specifically, in the inner-level, DROP decomposes offline data into multiple subsets, and learns an MBO score model (a1). To keep safe exploitation to the score model in the outer-level, we explicitly learn a behavior embedding and introduce a conservative regularization (a2). During testing, we show that DROP permits deployment adaptation, enabling an adaptive inference across states (a3). Empirically, we evaluate DROP on various tasks, showing that DROP gains comparable or better performance compared to prior methods.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 66 of its 76 references that are in the map. First posted on arXiv 26 Jun 2023.

Builds on

66 of 76 references.

Closest ancestors · 33

Same direction, newest first

  1. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning22 Jun
  2. 2023Beyond Reward: Offline Preference-guided Policy Optimization25 May
  3. 2023Behavior Proximal Policy Optimization22 Feb
  4. 2022Offline RL Policies Should be Trained to be Adaptive5 Jul
  5. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learning26 Apr
  6. 2022DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning13 Mar
  7. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  8. 2021RvS: What is Essential for Offline RL via Supervised Learning?20 Dec
  9. 2021Dealing with the Unknown: Pessimistic Offline Reinforcement Learning9 Nov
  10. 2021Offline Reinforcement Learning with Value-based Episodic Memory19 Oct
  11. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  12. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
21 more
  1. 2021BRAC+: Improved Behavior Regularized Actor Critic for Offline Reinforcement Learning2 Oct
  2. 2021Conservative Offline Distributional Reinforcement Learning12 Jul
  3. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation21 Jun
  4. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL16 Jun
  5. 2021Offline RL Without Off-Policy Evaluation16 Jun
  6. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  7. 2021Offline Reinforcement Learning as Anti-Exploration11 Jun
  8. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  9. 2021Regularized Behavior Value Estimation17 Mar
  10. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  11. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  12. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  13. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  14. 2020Critic Regularized Regression26 Jun
  15. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  16. 2020MOPO: Model-based Offline Policy Optimization27 May
  17. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  18. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  19. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  20. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  21. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 33

  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  3. 2017Hindsight Experience Replaycited by 1,136
  4. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  6. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  7. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  8. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  9. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  10. 2018Maximum a Posteriori Policy Optimisationcited by 328
  11. 2019A Theory of Regularized Markov Decision Processescited by 286
  12. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
21 more
  1. 2019Goal-conditioned Imitation Learningcited by 162
  2. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  3. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  4. 2019Conditioning by adaptive sampling for robust designcited by 111
  5. 2019Training Agents using Upside-Down Reinforcement Learningcited by 107
  6. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 106
  7. 2019Reward-Conditioned Policiescited by 102
  8. 2022Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimizationcited by 97
  9. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 91
  10. 2018Temporal Difference Models: Model-Free Deep RL for Model-Based Controlcited by 89
  11. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  12. 2021Conservative Objective Models for Effective Offline Model-Based Optimizationcited by 88
  13. 2019Model Inversion Networks for Model-Based Optimizationcited by 83
  14. 2020Rewriting History with Inverse RL: Hindsight Inference for Policy Improvementcited by 60
  15. 2020Autofocused oracles for model-based designcited by 50
  16. 2021RoMA: Robust Model Adaptation for Offline Model-based Optimizationcited by 39
  17. 2021Offline Model-Based Optimization via Normalized Maximum Likelihood Estimationcited by 38
  18. 2021Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learningcited by 17
  19. 2021Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learningcited by 15
  20. 2022Switch Trajectory Transformer with Distributional Value Approximation for Multi-Task Reinforcement Learningcited by 11
  21. 2021Hindsight Foresight Relabeling for Meta-Reinforcement Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 3; 2026: 2.

8 papers.

  1. 2024Reinformer: Max-Return Sequence Modeling for Offline RLcited by 25
  2. 2025Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representationscited by 5
  3. 2024DIDI: Diffusion-Guided Diversity for Offline Behavioral Generationcited by 4
  4. 2024Sample-efficient Unsupervised Policy Cloning from Ensemble Self-supervised Labeled Videoscited by 4
  5. 2025RL-Selector: Reinforcement Learning-Guided Data Selection via Redundancy Assessmentcited by 3
  6. 2025Test-Time Policy Adaptation for Enhanced Multi-Turn Interactions with LLMscited by 3
  7. 2026Words & Weights: Streamlining Multi-Turn Interactions via Co-Adaptationcited by 1
  8. 2026From Static Policies to Adaptive Priors in Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Bi-Level Offline Policy Optimization with Limited Exploration
  2. 2025Model Selection for Off-policy Evaluation: New Algorithms and Experimental Protocol
  3. 2022Conservative Bayesian Model-Based Value Expansion for Offline Policy Optimization
  4. 2025Behaviour Policy Optimization: Provably Lower Variance Return Estimates for Off-Policy Reinforcement Learning
  5. 2023Offline Policy Evaluation for Reinforcement Learning with Adaptively Collected Data
  6. 2023On the Sample Complexity of Vanilla Model-Based Offline Reinforcement Learning with Dependent Samples
  7. 2020Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation
  8. 2025Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspective