mlsanityData to 14 Oct 2026

Policy Finetuning in Reinforcement Learning via Design of Experiments using Offline Data

arXiv 2307.04354 ↗First posted 10 Jul 2023cs.LG stat.MLcited by 11

Not on the map. Why?

Abstract (arXiv)

In some applications of reinforcement learning, a dataset of pre-collected experience is already available but it is also possible to acquire some additional online data to help improve the quality of the policy. However, it may be preferable to gather additional data with a single, non-reactive exploration policy and avoid the engineering costs associated with switching policies. In this paper we propose an algorithm with provable guarantees that can leverage an offline dataset to design a single non-reactive policy for exploration. We theoretically analyze the algorithm and measure the quality of the final policy as a function of the local coverage of the original dataset and the amount of additional data collected.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 67 of its 96 references that are in the map. First posted on arXiv 10 Jul 2023.

Builds on

67 of 96 references.

References · 67

  1. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  2. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  3. 2017Minimax Regret Bounds for Reinforcement Learningcited by 503
  4. 2018Is Q-learning Provably Efficient?cited by 487
  5. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  6. 2019Reinforcement Learning in Healthcare: A Surveycited by 310
  7. 2019Information-Theoretic Considerations in Batch Reinforcement Learningcited by 287
  8. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  9. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  10. 2018Provably Efficient Maximum Entropy Explorationcited by 242
  11. 2019Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Boundscited by 195
  12. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
38 more
  1. 2021Reinforcement learning based recommender systems: A surveycited by 188
  2. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  3. 2017Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learningcited by 176
  4. 2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
  5. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 164
  6. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 158
  7. 2020What are the Statistical Limits of Offline RL with Linear Function Approximation?cited by 136
  8. 2020Almost Optimal Model-Free Reinforcement Learning via Reference-Advantage Decompositioncited by 127
  9. 2021Provably Efficient Reinforcement Learning with Linear Function Approximation Under Adaptivity Constraintscited by 124
  10. 2022Offline Reinforcement Learning with Realizability and Single-policy Concentrabilitycited by 123
  11. 2019Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learningcited by 114
  12. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learningcited by 110
  13. 2015Sample Complexity of Episodic Fixed-Horizon Reinforcement Learningcited by 106
  14. 2020Batch Value-function Approximation with Only Realizabilitycited by 104
  15. 2019Batched Multi-armed Bandits Problemcited by 98
  16. 2020Fast active learning for pure exploration in reinforcement learningcited by 92
  17. 2020On Reward-Free Reinforcement Learning with Linear Function Approximationcited by 89
  18. 2018Policy Certificates: Towards Accountable Reinforcement Learningcited by 81
  19. 2019Provably Efficient Q-Learning with Low Switching Costcited by 78
  20. 2020Adaptive Reward-Free Explorationcited by 77
  21. 2020Provably Good Batch Reinforcement Learning Without Great Explorationcited by 77
  22. 2019Efficiently Breaking the Curse of Horizon in Off-Policy Evaluation with Double Reinforcement Learningcited by 75
  23. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximationcited by 71
  24. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimismcited by 65
  25. 2013Online Learning with Switching Costs and Other Adaptive Adversariescited by 61
  26. 2018Evaluating Reinforcement Learning Algorithms in Observational Health Settingscited by 60
  27. 2020Asymptotically Efficient Off-Policy Evaluation for Tabular Reinforcement Learningcited by 60
  28. 2020Provably Efficient Reward-Agnostic Navigation with Linear Value Iterationcited by 56
  29. 2022Nearly Minimax Optimal Offline Reinforcement Learning with Linear Function Approximation: Single-Agent MDP and Markov Gamecited by 55
  30. 2020Linear Bandits with Limited Adaptivity and Learning Distributional Optimal Designcited by 51
  31. 2022Reward-Free RL is No Harder Than Reward-Aware RL in Linear Markov Decision Processescited by 49
  32. 2018Variance-Aware Regret Bounds for Undiscounted Reinforcement Learning in MDPscited by 43
  33. 2021Risk Bounds and Rademacher Complexity in Batch Reinforcement Learningcited by 42
  34. 2018Experience-driven Networking: A Deep Reinforcement Learning based Approachcited by 41
  35. 2021A Provably Efficient Algorithm for Linear Markov Decision Process with Low Switching Costcited by 40
  36. 2020Task-agnostic Exploration in Reinforcement Learningcited by 37
  37. 2020Planning in Markov Decision Processes with Gap-Dependent Sample Complexitycited by 34
  38. 2022Sample-Efficient Reinforcement Learning with loglog(T) Switching Costcited by 32

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 4; 2025: 5; 2026: 1.

11 papers.

  1. 2024Hybrid Transfer Reinforcement Learning: Provable Sample Efficiency from Shifted-Dynamics Datacited by 8
  2. 2024OLLIE: Imitation Learning from Offline Pretraining to Online Finetuningcited by 6
  3. 2024Hybrid Reinforcement Learning from Offline Observation Alonecited by 5
  4. 2025Best Policy Learning from Trajectory Preference Feedbackcited by 5
  5. 2023Sample-Efficiency in Multi-Batch Reinforcement Learning: The Need for Dimension-Dependent Adaptivitycited by 2
  6. 2024Multiple-policy Evaluation via Density Estimationcited by 2
  7. 2025On The Statistical Complexity of Offline Decision-Makingcited by 2
  8. 2025Improving the Data-efficiency of Reinforcement Learning by Warm-starting with LLMcited by 2
  9. 2025Behavior-Adaptive Q-Learning: A Unifying Framework for Offline-to-Online RLcited by 1
  10. 2025From Static Constraints to Dynamic Adaptation: Sample-Level Constraint Relaxation for Offline-to-Online Reinforcement Learningcited by 0
  11. 2026Escaping Offline Pessimism: Vector-Field Reward Shaping for Safe Frontier Explorationcited by 0