mlsanityData to 14 Oct 2026

On Value Functions and the Agent-Environment Boundary

arXiv 1905.13341 ↗First posted 30 May 2019cs.LG cs.AI stat.MLcited by 19

Not on the map. Why?

Abstract (arXiv)

When function approximation is deployed in reinforcement learning (RL), the same problem may be formulated in different ways, often by treating a pre-processing step as a part of the environment or as part of the agent. As a consequence, fundamental concepts in RL, such as (optimal) value functions, are not uniquely defined as they depend on where we draw this agent-environment boundary, causing problems in theoretical analyses that provide optimality guarantees. We address this issue via a simple and novel boundary-invariant analysis of Fitted Q-Iteration, a representative RL algorithm, where the assumptions and the guarantees are invariant to the choice of boundary. We also discuss closely related issues on state resetting and Monte-Carlo Tree Search, deterministic vs stochastic systems, imitation learning, and the verifiability of theoretical assumptions from data.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). First posted on arXiv 30 May 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 2; 2022: 4; 2023: 4; 2024: 2; 2025: 6; 2026: 1.

19 papers.

Sort:

By area:

  1. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimismcited by 279
  2. 2022Jump-Start Reinforcement Learningcited by 112
  3. 2021Risk Bounds and Rademacher Complexity in Batch Reinforcement Learningcited by 42
  4. 2022Optimal Conservative Offline RL with General Function Approximation via Augmented Lagrangiancited by 29
  5. 2024Three Dogmas of Reinforcement Learningcited by 14
  6. 2022Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Rewardcited by 13
  7. 2023Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learningcited by 13
  8. 2023Provably Efficient Offline Goal-Conditioned Reinforcement Learning with General Function Approximation and Single-Policy Concentrabilitycited by 5
  9. 2025Offline Learning for Combinatorial Multi-armed Banditscited by 5
  10. 2025Agency Is Frame-Dependentcited by 5
  11. 2023Achieving the Asymptotically Optimal Sample Complexity of Offline Reinforcement Learning: A DRO-Based Approachcited by 3
  12. 2024A Black Swan Hypothesis: The Role of Human Irrationality in AI Safetycited by 3
7 more
  1. 2022Build generally reusable agent-environment interaction modelscited by 1
  2. 2023Neural Network Approximation for Pessimistic Offline Reinforcement Learningcited by 1
  3. 2025Real-World Reinforcement Learning of Active Perception Behaviorscited by 1
  4. 2025The World Is Bigger! A Computationally-Embedded Perspective on the Big World Hypothesiscited by 1
  5. 2025Distributions as Actions: A Unified Framework for Diverse Action Spacescited by 0
  6. 2025Selecting Belief-State Approximations in Simulators with Latent Statescited by 0
  7. 2026Expanding Spatial and Temporal Context for Robotic Imitation Learning With Scene Graphscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Offline Reinforcement Learning: Fundamental Barriers for Value Function Approximation
  2. 2025Inferring Transition Dynamics from Value Functions
  3. 2025The Role of Environment Access in Agnostic Reinforcement Learning
  4. 2026Mathematical methods of reinforcement learning
  5. 2025Agnostic Reinforcement Learning: Foundations and Algorithms
  6. 2024Taming "data-hungry" reinforcement learning? Stability in continuous state-action spaces
  7. 2026Reinforcement Learning with Function Approximation for Non-Markov Processes
  8. 2024On the continuity and smoothness of the value function in reinforcement learning and optimal control