SAINT: Attention-Based Policies for Discrete Combinatorial Action Spaces

arXiv 2505.12109 ↗First posted 17 May 2025cs.LG cs.AIcited by 0

On the map: Deep reinforcement learning methods → RL with diffusion and flow models

Abstract (arXiv)

The combinatorial structure of many real-world action spaces leads to exponential growth in the number of possible actions, limiting the effectiveness of conventional reinforcement learning algorithms. Recent approaches for combinatorial action spaces impose factorized or sequential structures over sub-actions, failing to capture complex joint behavior. We introduce the Sub-Action Interaction Network using Transformers (SAINT), a novel policy architecture that represents multi-component actions as unordered sets and models their dependencies via self-attention conditioned on the global state. SAINT is permutation-invariant, sample-efficient, and compatible with standard policy optimization algorithms. In 18 distinct combinatorial environments across three task domains, including environments with 1.35 × 1018 possible actions, SAINT consistently outperforms strong baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in RL with diffusion and flow models, one of 15 directions in Deep reinforcement learning methods. Builds on 46 of its 58 references that are in the map. First posted on arXiv 17 May 2025.

Builds on

46 of 58 references.

References · 46

2017Attention Is All You Needcited by 60,361
2017Proximal Policy Optimization Algorithmscited by 15,215
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
2015Continuous control with deep reinforcement learningcited by 3,407
2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,333
2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
2019Generating Long Sequences with Sparse Transformerscited by 1,546
2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546

34 more

2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
2018DeepMind Control Suitecited by 956
2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
2025FAST: Efficient Action Tokenization for Vision-Language-Action Modelscited by 665
2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
2016Neural Combinatorial Optimization with Reinforcement Learningcited by 608
2020Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learningcited by 485
2018Reinforcement Learning for Solving the Vehicle Routing Problemcited by 333
2022Transformers are Sample-Efficient World Modelscited by 328
2018Actor-Attention-Critic for Multi-Agent Reinforcement Learningcited by 261
2018Distributed Distributional Deterministic Policy Gradientscited by 199
2022Masked World Models for Visual Controlcited by 170
2015Deep Reinforcement Learning in Large Discrete Action Spacescited by 161
2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 131
2019CityFlow: A Multi-Agent Reinforcement Learning Environment for Large Scale City Traffic Scenariocited by 115
2017Action Branching Architectures for Deep Reinforcement Learningcited by 92
2015Deep Reinforcement Learning with a Natural Language Action Spacecited by 88
2018Learn What Not to Learn: Action Elimination with Deep Reinforcement Learningcited by 77
2018Relational Deep Reinforcement Learningcited by 73
2019Discretizing Continuous Action Space for On-Policy Optimizationcited by 68
2020Reinforcement Learning with Combinatorial Actions: An Application to Vehicle Routingcited by 61
2017Discrete Sequential Prediction of Continuous Actions for Deep RLcited by 51
2023Leveraging Factored Action Spaces for Efficient Offline Reinforcement Learning in Healthcarecited by 42
2022Deep Transformer Q-Networks for Partially Observable Reinforcement Learningcited by 36
2020Q-Learning in enormous action spaces via amortized approximate maximizationcited by 30
2019Growing Action Spacescited by 29
2021StARformer: Transformer with State-Action-Reward Representations for Visual Reinforcement Learningcited by 27
2023Generative Modelling of Stochastic Actions with Arbitrary Constraints in Reinforcement Learningcited by 10
2018Efficient Entropy for Policy Gradient with Multidimensional Action Spacecited by 5
2021Vision Transformer for Learning Driving Policies in Complex Multi-Agent Environmentscited by 5
2024An Investigation of Offline Reinforcement Learning in Factorisable Action Spacescited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way