mlsanityData to 14 Oct 2026

TensorPlan and the Few Actions Lower Bound for Planning in MDPs under Linear Realizability of Optimal Value Functions

arXiv 2110.02195 ↗First posted 5 Oct 2021cs.LG cs.AI stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

We consider the minimax query complexity of online planning with a generative model in fixed-horizon Markov decision processes (MDPs) with linear function approximation. Following recent works, we consider broad classes of problems where either (i) the optimal value function v^star or (ii) the optimal action-value function q^star lie in the linear span of some features; or (iii) both v^star and q^star lie in the linear span when restricted to the states reachable from the starting state. Recently, Weisz et al. (2021b) showed that under (ii) the minimax query complexity of any planning algorithm is at least exponential in the horizon H or in the feature dimension d when the size A of the action set can be chosen to be exponential in min(d,H). On the other hand, for the setting (i), Weisz et al. (2021a) introduced TensorPlan, a planner whose query cost is polynomial in all relevant quantities when the number of actions is fixed. Among other things, these two works left open the question whether polynomial query complexity is possible when A is subexponential in min(d,H). In this paper we answer this question in the negative: we show that an exponentially large lower bound holds when A=Omega(min(d1/4,H1/2)), under either (i), (ii) or (iii). In particular, this implies a perhaps surprising exponential separation of query complexity compared to the work of Du et al. (2021) who prove a polynomial upper bound when (iii) holds for all states. Furthermore, we show that the upper bound of TensorPlan can be extended to hold under (iii) and, for MDPs with deterministic transitions and stochastic rewards, also under (ii).

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 7 of its 23 references that are in the map. First posted on arXiv 5 Oct 2021.

Builds on

7 of 23 references.

References · 7

  1. 2021Bilinear Classes: A Structural Framework for Provable Generalization in RLcited by 178
  2. 2019Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?cited by 140
  3. 2019Learning with Good Feature Representations in Bandits and in RL with a Generative Modelcited by 127
  4. 2020Exponential Lower Bounds for Planning in MDPs With Linearly-Realizable Optimal Action-Value Functionscited by 71
  5. 2021An Exponential Lower Bound for Linearly-Realizable MDPs with Constant Suboptimality Gapcited by 40
  6. 2013Improved and Generalized Upper Bounds on the Complexity of Policy Iterationcited by 21
  7. 2021On Query-efficient Planning in MDPs under Linear Realizability of the Optimal State-value Functioncited by 18

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 1; 2024: 1.

7 papers.

  1. 2022Computationally Efficient PAC RL in POMDPs with Latent Determinism and Conditional Embeddingscited by 12
  2. 2022A Few Expert Queries Suffices for Sample-Efficient RL with Resets and Linear Value Approximationcited by 7
  3. 2022Stabilizing Q-learning with Linear Architectures for Provably Efficient Learningcited by 5
  4. 2023Exponential Hardness of Reinforcement Learning with Linear Function Approximationcited by 4
  5. 2022Computational-Statistical Gaps in Reinforcement Learningcited by 3
  6. 2022Linear Reinforcement Learning with Ball Structure Action Spacecited by 1
  7. 2024Misspecified Q-Learning with Sparse Linear Function Approximation: Tight Bounds on Approximation Errorcited by 0