# Reinforcement Learning for Task Specifications with Action-Constraints

arXiv: https://arxiv.org/abs/2201.00286 · HTML: https://mlsanity.com/paper/2201.00286/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 0 of its 8 references that are in the map.
- First posted on arXiv 2 Jan 2022.

## Abstract (arXiv)

In this paper, we use concepts from supervisory control theory of discrete event systems to propose a method to learn optimal control policies for a finite-state Markov Decision Process (MDP) in which (only) certain sequences of actions are deemed unsafe (respectively safe). We assume that the set of action sequences that are deemed unsafe and/or safe are given in terms of a finite-state automaton; and propose a supervisor that disables a subset of actions at every state of the MDP so that the constraints on action sequence are satisfied. Then we present a version of the Q-learning algorithm for learning optimal policies in the presence of non-Markovian action-sequence and state constraints, where we use the development of reward machines to handle the state constraints. We illustrate the method using an example that captures the utility of automata-based methods for non-Markovian state and action specifications for reinforcement learning and show the results of simulations in this setting.