mlsanityData to 7 Oct 2026

Improving and Accelerating Offline RL in Large Discrete Action Spaces with Structured Policy Initialization

arXiv 2601.04441 ↗First posted 7 Jan 2026cs.LGcited by 0

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Reinforcement learning in discrete combinatorial action spaces requires searching over exponentially many joint actions to simultaneously select multiple sub-actions that form coherent combinations. Existing approaches either simplify policy learning by assuming independence across sub-actions, which often yields incoherent or invalid actions, or attempt to learn action structure and control jointly, which is slow and unstable. We introduce Structured Policy Initialization (SPIN), a two-stage framework that first pre-trains an Action Structure Model (ASM) to capture the manifold of valid actions, then freezes this representation and trains lightweight policy heads for control. On challenging discrete DM Control benchmarks, SPIN improves average return by up to 39% over the state of the art while reducing time to convergence by up to 12.8×.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 42 of its 53 references that are in the map. First posted on arXiv 7 Jan 2026.

Builds on

42 of 53 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024An Investigation of Offline Reinforcement Learning in Factorisable Action Spaces17 Nov
  2. 2024BraVE: Offline Reinforcement Learning for Discrete Combinatorial Action Spaces28 Oct
  3. 2023Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcare2 May
  4. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  6. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  6. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,777
  7. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  8. 2018World Modelscited by 1,444
  9. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  10. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  11. 2018DeepMind Control Suitecited by 935
  12. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
24 more
  1. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  2. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  3. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  4. 2018Reinforcement Learning for Solving the Vehicle Routing Problemcited by 327
  5. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  6. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  7. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  8. 2015Deep Reinforcement Learning in Large Discrete Action Spacescited by 158
  9. 2021APS: Active Pretraining with Successor Featurescited by 109
  10. 2018Deep Reinforcement Learning for Page-wise Recommendationscited by 99
  11. 2017Action Branching Architectures for Deep Reinforcement Learningcited by 90
  12. 2016Loss is its own Reward: Self-Supervision for Reinforcement Learningcited by 72
  13. 2020Reinforcement Learning with Combinatorial Actions: An Application to Vehicle Routingcited by 61
  14. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 51
  15. 2020Deep Visual Reasoning: Learning to Predict Action Sequences for Task and Motion Planning from an Initial Scene Imagecited by 45
  16. 2022Masked Autoencoding for Scalable and Generalizable Decision Makingcited by 44
  17. 2023SMART: Self-supervised Multi-task pretrAining with contRol Transformerscited by 44
  18. 2023Masked Trajectory Models for Prediction, Representation, and Controlcited by 40
  19. 2020Q-Learning in enormous action spaces via amortized approximate maximizationcited by 30
  20. 2019Growing Action Spacescited by 29
  21. 2025Fast Adaptation with Behavioral Foundation Modelscited by 20
  22. 2023Generative Modelling of Stochastic Actions with Arbitrary Constraints in Reinforcement Learningcited by 10
  23. 2018Efficient Entropy for Policy Gradient with Multidimensional Action Spacecited by 5
  24. 2023RePreM: Representation Pre-training with Masked Model for Reinforcement Learningcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spaces
  2. 2025Structured Reinforcement Learning for Combinatorial Decision-Making
  3. 2026Latent Spherical Flow Policy for Reinforcement Learning with Combinatorial Actions
  4. 2026Flow Matching for Offline Reinforcement Learning with Discrete Actions
  5. 2025Scalable Offline Model-Based RL with Action Chunks
  6. 2023Structured State Space Models for In-Context Reinforcement Learning
  7. 2023Diverse Policy Optimization for Structured Action Space
  8. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning