mlsanityData to 7 Oct 2026

Model-Based Reinforcement Learning with Value-Targeted Regression

arXiv 2006.01107 ↗First posted 1 Jun 2020cs.LG stat.MLcited by 233

Not on the map. Why?

Abstract (arXiv)

This paper studies model-based reinforcement learning (RL) for regret minimization. We focus on finite-horizon episodic RL where the transition model P belongs to a known family of models P, a special case of which is when models in P take the form of linear mixtures: Pθ = sumi=1d θiPi. We propose a model based RL algorithm that is based on optimism principle: In each episode, the set of models that are `consistent' with the data collected is constructed. The criterion of consistency is based on the total squared error of that the model incurs on the task of predicting values as determined by the last value estimate along the transitions. The next value function is then chosen by solving the optimistic planning problem with the constructed set of models. We derive a bound on the regret, which, in the special case of linear mixtures, the regret bound takes the form tildeO(d√H3T), where H, T and d are the horizon, total number of steps and dimension of θ, respectively. In particular, this regret bound is independent of the total number of states or actions, and is close to a lower bound Omega(√HdT). For a general model family P, the regret bound is derived using the notion of the so-called Eluder dimension proposed by Russo & Van Roy (2014).

Key facts. Cited by 233 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Jun 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 56; 2022: 57; 2023: 59; 2024: 32; 2025: 13; 2026: 15.

233 papers.

Sort:

By area:

  1. 2020Is Pessimism Provably Efficient for Offline RL?cited by 362
  2. 2021Bilinear Classes: A Structural Framework for Provable Generalization in RLcited by 176
  3. 2021Pessimistic Model-based Offline Reinforcement Learning under Partial Coveragecited by 160
  4. 2021Provably Efficient Reinforcement Learning with Linear Function Approximation Under Adaptivity Constraintscited by 124
  5. 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
  6. 2021Model-free Representation Learning and Exploration in Low-rank MDPscited by 82
  7. 2022Human-in-the-loop: Provably Efficient Preference-based Reinforcement Learning with General Function Approximationcited by 80
  8. 2025Outcome-based Exploration for LLM Reasoningcited by 79
  9. 2021Understanding Domain Randomization for Sim-to-real Transfercited by 69
  10. 2022Near-optimal Offline Reinforcement Learning with Linear Representation: Leveraging Variance Information with Pessimismcited by 63
  11. 2024Active Preference Optimization for Sample Efficient RLHFcited by 61
  12. 2022Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexitycited by 59
38 more
  1. 2022Nearly Minimax Optimal Reinforcement Learning for Linear Markov Decision Processescited by 57
  2. 2021Cautiously Optimistic Policy Optimization and Exploration with Linear Function Approximationcited by 53
  3. 2022Computationally Efficient Horizon-Free Reinforcement Learning for Linear Mixture MDPscited by 52
  4. 2021Randomized Exploration for Reinforcement Learning with General Value Function Approximationcited by 49
  5. 2022Reward-Free RL is No Harder Than Reward-Aware RL in Linear Markov Decision Processescited by 49
  6. 2023Reason for Future, Act for Now: A Principled Framework for Autonomous LLM Agents with Provable Sample Efficiencycited by 46
  7. 2021Improved Variance-Aware Confidence Sets for Linear Bandits and Linear Mixture MDPcited by 45
  8. 2021Near-optimal Representation Learning for Linear Bandits and Linear RLcited by 45
  9. 2022Leveraging Offline Data in Online Reinforcement Learningcited by 42
  10. 2021An Exponential Lower Bound for Linearly-Realizable MDPs with Constant Suboptimality Gapcited by 40
  11. 2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
  12. 2023Variance-Dependent Regret Bounds for Linear Bandits and Reinforcement Learning: Adaptivity and Computational Efficiencycited by 39
  13. 2021A Provably Efficient Algorithm for Linear Markov Decision Process with Low Switching Costcited by 38
  14. 2021First-Order Regret in Reinforcement Learning with Linear Function Approximation: A Robust Estimation Approachcited by 38
  15. 2022Provable Benefits of Representational Transfer in Reinforcement Learningcited by 38
  16. 2021Control-Oriented Model-Based Reinforcement Learning with Implicit Differentiationcited by 35
  17. 2021Deep reinforcement learning in medical imaging: A literature reviewcited by 34
  18. 2021Variance-Aware Off-Policy Evaluation with Linear Function Approximationcited by 34
  19. 2021Proper Value Equivalencecited by 33
  20. 2022Instance-Dependent Near-Optimal Policy Identification in Linear MDPs via Online Experiment Designcited by 33
  21. 2021Provable Model-based Nonlinear Bandit and Reinforcement Learning: Shelve Optimism, Embrace Virtual Curvaturecited by 32
  22. 2022On the Complexity of Adversarial Decision Makingcited by 32
  23. 2022Value Gradient weighted Model-Based Reinforcement Learningcited by 31
  24. 2022Model-based RL with Optimistic Posterior Sampling: Structural Conditions and Sample Complexitycited by 31
  25. 2022A General Framework for Sample-Efficient Function Approximation in Reinforcement Learningcited by 31
  26. 2021Robust Policy Gradient against Strong Data Corruptioncited by 30
  27. 2022Nearly Minimax Optimal Reinforcement Learning with Linear Function Approximationcited by 30
  28. 2021Reward-Free Model-Based Reinforcement Learning with Linear Function Approximationcited by 29
  29. 2022Provably Efficient Model-Free Constrained RL with Linear Function Approximationcited by 29
  30. 2023Tight Guarantees for Interactive Decision Making with the Decision-Estimation Coefficientcited by 29
  31. 2023Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlocited by 29
  32. 2021Near-optimal Policy Optimization Algorithms for Learning Adversarial Linear Mixture MDPscited by 28
  33. 2023A Theoretical Analysis of Optimistic Proximal Policy Optimization in Linear Markov Decision Processescited by 28
  34. 2021Sample-Efficient Reinforcement Learning Is Feasible for Linearly Realizable MDPs with Limited Revisitingcited by 27
  35. 2022Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learningcited by 26
  36. 2021Almost Optimal Algorithms for Two-player Zero-Sum Linear Mixture Markov Gamescited by 25
  37. 2022Proximal Point Imitation Learningcited by 25
  38. 2021Learning Representations for Pixel-based Control: What Matters and Why?cited by 24

50 most cited of 233 All 233 (JSON)