mlsanityData to 7 Oct 2026

Experiment Planning with Function Approximation

arXiv 2401.05193 ↗First posted 10 Jan 2024cs.LG cs.AI stat.MLcited by 5

Not on the map. Why?

Abstract (arXiv)

We study the problem of experiment planning with function approximation in contextual bandit problems. In settings where there is a significant overhead to deploying adaptive algorithms -- for example, when the execution of the data collection policies is required to be distributed, or a human in the loop is needed to implement these policies -- producing in advance a set of policies for data collection is paramount. We study the setting where a large dataset of contexts but not rewards is available and may be used by the learner to design an effective data collection strategy. Although when rewards are linear this problem has been well studied, results are still missing for more complex reward models. In this work we propose two experiment planning strategies compatible with function approximation. The first is an eluder planning and sampling procedure that can recover optimality guarantees depending on the eluder dimension of the reward function class. For the second, we show that a uniform sampler achieves competitive optimality rates in the setting where the number of actions is small. We finalize our results introducing a statistical gap fleshing out the fundamental differences between planning and adaptive learning and provide results for planning with model selection.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 33 of its 54 references that are in the map. First posted on arXiv 10 Jan 2024.

Builds on

33 of 54 references.

References · 33

  1. 2015Continuous control with deep reinforcement learningcited by 3,279
  2. 2016Deep Reinforcement Learning for Dialogue Generationcited by 286
  3. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 222
  4. 2014Taming the Monster: A Fast and Simple Algorithm for Contextual Banditscited by 200
  5. 2020Beyond UCB: Optimal and Efficient Contextual Bandits with Regression Oraclescited by 195
  6. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  7. 2021Bilinear Classes: A Structural Framework for Provable Generalization in RLcited by 176
  8. 2019Learning with Good Feature Representations in Bandits and in RL with a Generative Modelcited by 124
  9. 2014Model-based Reinforcement Learning and the Eluder Dimensioncited by 116
  10. 2020Bypassing the Monster: A Faster and Simpler Optimal Algorithm for Contextual Bandits under Realizabilitycited by 114
  11. 2016Corralling a Band of Bandit Algorithmscited by 99
  12. 2014Best-Arm Identification in Linear Banditscited by 98
21 more
  1. 2018Practical Contextual Bandits with Regression Oraclescited by 92
  2. 2020On Reward-Free Reinforcement Learning with Linear Function Approximationcited by 89
  3. 2019Sequential Experimental Design for Transductive Linear Banditscited by 88
  4. 2020Model Selection in Contextual Stochastic Bandit Problemscited by 71
  5. 2020Optimal Best-arm Identification in Linear Banditscited by 70
  6. 2020Gamification of Pure Exploration for Linear Banditscited by 61
  7. 2020Exponential Lower Bounds for Batch Reinforcement Learning: Batch RL can be Exponentially Harder than Online RLcited by 55
  8. 2020Linear Bandits with Limited Adaptivity and Learning Distributional Optimal Designcited by 51
  9. 2019Regret Bounds for Batched Banditscited by 49
  10. 2020Regret Bound Balancing and Elimination for Model Selection in Bandits and RLcited by 45
  11. 2017Fully adaptive algorithm for pure exploration in linear banditscited by 43
  12. 2020Online Model Selection for Reinforcement Learning with Function Approximationcited by 28
  13. 2022On the Statistical Efficiency of Reward-Free Exploration in Non-Linear RLcited by 27
  14. 2021Understanding the Eluder Dimensioncited by 19
  15. 2021Design of Experiments for Stochastic Contextual Linear Banditscited by 18
  16. 2022Instance-optimal PAC Algorithms for Contextual Banditscited by 16
  17. 2014Bounded Regret for Finite-Armed Structured Banditscited by 15
  18. 2022Safe Exploration for Efficient Policy Evaluation and Comparisoncited by 15
  19. 2020Crush Optimism with Pessimism: Structured Bandits Beyond Asymptotic Optimalitycited by 7
  20. 2022Neural Design for Genetic Perturbation Experimentscited by 6
  21. 2021Parallelizing Contextual Banditscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 0; 2026: 2.

5 papers.

  1. 2024Second Order Bounds for Contextual Bandits with Function Approximationcited by 9
  2. 2026Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directionscited by 5
  3. 2024Active Few-Shot Fine-Tuningcited by 2
  4. 2024The Fallacy of Minimizing Cumulative Regret in the Sequential Task Settingcited by 1
  5. 2026Bayesian Online Model Selectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2020Batch Value-function Approximation with Only Realizability
  2. 2023Reward Learning as Doubly Nonparametric Bandits: Optimal Design and Scaling Laws
  3. 2023Sample-Efficiency in Multi-Batch Reinforcement Learning: The Need for Dimension-Dependent Adaptivity
  4. 2021Learning from an Exploring Demonstrator: Optimal Reward Estimation for Bandits
  5. 2023Contextual Bandits and Imitation Learning via Preference-Based Active Queries
  6. 2025Statistical Inference for Misspecified Contextual Bandits
  7. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  8. 2023Offline Reinforcement Learning with Additional Covering Distributions