# Timing is Everything: Learning to Act Selectively with Costly Actions and Budgetary Constraints

arXiv: https://arxiv.org/abs/2205.15953 · HTML: https://mlsanity.com/paper/2205.15953/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 41 references that are in the map.
- First posted on arXiv 31 May 2022.

## Builds on (most cited first, 12 of 12)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentation](https://mlsanity.com/paper/2202.06558/) (2022)
- [Discovering Options for Exploration by Minimizing Cover Time](https://mlsanity.com/paper/1903.00606/) (2019)
- [Flexible Option Learning](https://mlsanity.com/paper/2112.03097/) (2021)
- [Deep Reinforcement Learning](https://mlsanity.com/paper/1810.06339/) (2018)
- [Deep Reinforcement Learning for Optimal Stopping with Application in Financial Engineering](https://mlsanity.com/paper/2105.08877/) (2021)
- [A Viscosity Approach to Stochastic Differential Games of Control and Stopping Involving Impulsive Control](https://mlsanity.com/paper/1803.11432/) (2018)
- [Cutting Your Losses: Learning Fault-Tolerant Control and Optimal Stopping under Adverse Risk](https://mlsanity.com/paper/1902.05045/) (2019)
- [Optimal Selection of Transaction Costs in a Dynamic Principal-Agent Problem](https://mlsanity.com/paper/1805.01062/) (2018)
- [Optimal Capital Injections with the Risk of Ruin: A Stochastic Differential Game of Impulse Control and Stopping Approach](https://mlsanity.com/paper/1805.01578/) (2018)
- [Duopoly Investment Problems with Minimally Bounded Adjustment Costs](https://mlsanity.com/paper/1805.11974/) (2018)

## Built on by (most cited first, 2 of 2)

- [All Language Models Large and Small](https://mlsanity.com/paper/2402.12061/) (2024)
- [Learning Multi-Timescale Interventions under Safety and Resource Constraints](https://mlsanity.com/paper/2508.03875/) (2025)

## Abstract (arXiv)

Many real-world settings involve costs for performing actions; transaction costs in financial systems and fuel costs being common examples. In these settings, performing actions at each time step quickly accumulates costs leading to vastly suboptimal outcomes. Additionally, repeatedly acting produces wear and tear and ultimately, damage. Determining textitwhen to act is crucial for achieving successful outcomes and yet, the challenge of efficiently textitlearning to behave optimally when actions incur minimally bounded costs remains unresolved. In this paper, we introduce a reinforcement learning (RL) framework named Learnable Impulse Control Reinforcement Algorithm (LICRA), for learning to optimally select both when to act and which actions to take when actions incur costs. At the core of LICRA is a nested structure that combines RL and a form of policy known as textitimpulse control which learns to maximise objectives when actions incur costs. We prove that LICRA, which seamlessly adopts any RL method, converges to policies that optimally select when to perform actions and their optimal magnitudes. We then augment LICRA to handle problems in which the agent can perform at most k<∞ actions and more generally, faces a budget constraint. We show LICRA learns the optimal value function and ensures budget constraints are satisfied almost surely. We demonstrate empirically LICRA's superior performance against benchmark RL methods in OpenAI gym's textitLunar Lander and in textitHighway environments and a variant of the Merton portfolio problem within finance.