# Action Pick-up in Dynamic Action Space Reinforcement Learning

arXiv: https://arxiv.org/abs/2304.00873 · HTML: https://mlsanity.com/paper/2304.00873/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 17 references that are in the map.
- First posted on arXiv 3 Apr 2023.

## Builds on (most cited first, 10 of 10)

- [Deep Reinforcement Learning in Large Discrete Action Spaces](https://mlsanity.com/paper/1512.07679/) (2015)
- [Reinforcement Learning for Non-Stationary Markov Decision Processes: The Blessing of (More) Optimism](https://mlsanity.com/paper/2006.14389/) (2020)
- [Learning Action Representations for Reinforcement Learning](https://mlsanity.com/paper/1902.00183/) (2019)
- [Optimizing for the Future in Non-Stationary MDPs](https://mlsanity.com/paper/2005.08158/) (2020)
- [Variational Regret Bounds for Reinforcement Learning](https://mlsanity.com/paper/1905.05857/) (2019)
- [A Sliding-Window Algorithm for Markov Decision Processes with Arbitrarily Changing Rewards and Transitions](https://mlsanity.com/paper/1805.10066/) (2018)
- [Generalization to New Actions in Reinforcement Learning](https://mlsanity.com/paper/2011.01928/) (2020)
- [Lifelong Learning with a Changing Action Set](https://mlsanity.com/paper/1906.01770/) (2019)
- [Learning Representations in Model-Free Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1810.10096/) (2018)
- [Planning and Learning with Stochastic Action Sets](https://mlsanity.com/paper/1805.02363/) (2018)

## Built on by (most cited first, 3 of 3)

- [In-Context Reinforcement Learning for Variable Action Spaces](https://mlsanity.com/paper/2312.13327/) (2023)
- [A Reinforcement Learning-Based Task Mapping Method to Improve the Reliability of Clustered Manycores](https://mlsanity.com/paper/2412.19340/) (2024)
- [Zero-Shot Action Generalization with Limited Observations](https://mlsanity.com/paper/2503.08867/) (2025)

## Abstract (arXiv)

Most reinforcement learning algorithms are based on a key assumption that Markov decision processes (MDPs) are stationary. However, non-stationary MDPs with dynamic action space are omnipresent in real-world scenarios. Yet problems of dynamic action space reinforcement learning have been studied by many previous works, how to choose valuable actions from new and unseen actions to improve learning efficiency remains unaddressed. To tackle this problem, we propose an intelligent Action Pick-up (AP) algorithm to autonomously choose valuable actions that are most likely to boost performance from a set of new actions. In this paper, we first theoretically analyze and find that a prior optimal policy plays an important role in action pick-up by providing useful knowledge and experience. Then, we design two different AP methods: frequency-based global method and state clustering-based local method, based on the prior optimal policy. Finally, we evaluate the AP on two simulated but challenging environments where action spaces vary over time. Experimental results demonstrate that our proposed AP has advantages over baselines in learning efficiency.