mlsanityData to 14 Oct 2026

BraVE: Offline Reinforcement Learning for Discrete Combinatorial Action Spaces

arXiv 2410.21151 ↗First posted 28 Oct 2024cs.LGcited by 3

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning in high-dimensional, discrete action spaces is challenging due to the exponential scaling of the joint action space with the number of sub-actions and the complexity of modeling sub-action dependencies. Existing methods either exhaustively evaluate the action space, making them computationally infeasible, or factorize Q-values, failing to represent joint sub-action effects. We propose Branch Value Estimation (BraVE), a value-based method that uses tree-structured action traversal to evaluate a linear number of joint actions while preserving dependency structure. BraVE outperforms prior offline RL methods by up to 20× in environments with over four million actions.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 22 of its 36 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

22 of 36 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare2 May
  2. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  3. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 17

  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  3. 2018DeepMind Control Suitecited by 956
  4. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  5. 2016Neural Combinatorial Optimization with Reinforcement Learningcited by 608
  6. 2018Reinforcement Learning for Solving the Vehicle Routing Problemcited by 333
  7. 2020A Closer Look at Invalid Action Masking in Policy Gradient Algorithmscited by 171
  8. 2015Deep Reinforcement Learning in Large Discrete Action Spacescited by 161
  9. 2017Action Branching Architectures for Deep Reinforcement Learningcited by 92
  10. 2015Deep Reinforcement Learning with a Natural Language Action Spacecited by 88
  11. 2018Learn What Not to Learn: Action Elimination with Deep Reinforcement Learningcited by 77
  12. 2020Reinforcement Learning with Combinatorial Actions: An Application to Vehicle Routingcited by 61
5 more
  1. 2020Q-Learning in enormous action spaces via amortized approximate maximizationcited by 30
  2. 2019Growing Action Spacescited by 29
  3. 2019ASAC: Active Sensing using Actor-Critic modelscited by 13
  4. 2023Generative Modelling of Stochastic Actions with Arbitrary Constraints in Reinforcement Learningcited by 10
  5. 2023Conditionally Optimistic Exploration for Cooperative Deep Multi-Agent Reinforcement Learningcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Beyond the Proxy: Trajectory-Distilled Guidance for Offline GFlowNet Trainingcited by 0
  2. 2026Improving and Accelerating Offline RL in Large Discrete Action Spaces with Structured Policy Initializationcited by 0
  3. 2026TabQL: In-Context Q-Learning with Tabular Foundation Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024An Investigation of Offline Reinforcement Learning in Factorisable Action Spaces
  2. 2025MOBODY: Model Based Off-Dynamics Offline Reinforcement Learning
  3. 2020Offline Reinforcement Learning Hands-On
  4. 2023Matrix Estimation for Offline Reinforcement Learning with Low-Rank Structure
  5. 2026Learning from the Unseen: Offline Reinforcement Learning with Hidden Actions
  6. 2025A Clean Slate for Offline Reinforcement Learning
  7. 2021Exploiting Action Impact Regularity and Exogenous State Variables for Offline Reinforcement Learning
  8. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies