# Policy Regularization for Legible Behavior

arXiv: https://arxiv.org/abs/2203.04303 · HTML: https://mlsanity.com/paper/2203.04303/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 35 references that are in the map.
- First posted on arXiv 8 Mar 2022.

## Builds on (most cited first, 15 of 15)

- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Behavior Regularized Offline Reinforcement Learning](https://mlsanity.com/paper/1911.11361/) (2019)
- [Techniques for Interpretable Machine Learning](https://mlsanity.com/paper/1808.00033/) (2018)
- [Offline Reinforcement Learning with Fisher Divergence Critic Regularization](https://mlsanity.com/paper/2103.08050/) (2021)
- [Explainable Machine Learning for Scientific Insights and Discoveries](https://mlsanity.com/paper/1905.08883/) (2019)
- [Explainable Reinforcement Learning: A Survey](https://mlsanity.com/paper/2005.06247/) (2020)
- [Playing hard exploration games by watching YouTube](https://mlsanity.com/paper/1805.11592/) (2018)
- [Towards Interpretable Reinforcement Learning Using Attention Augmented Agents](https://mlsanity.com/paper/1906.02500/) (2019)
- [Interpretability of machine learning based prediction models in healthcare](https://mlsanity.com/paper/2002.08596/) (2020)
- [Toward Interpretable Deep Reinforcement Learning with Linear Model U-Trees](https://mlsanity.com/paper/1807.05887/) (2018)
- [Regularizing Action Policies for Smooth Control with Reinforcement Learning](https://mlsanity.com/paper/2012.06644/) (2020)
- [Explicability? Legibility? Predictability? Transparency? Privacy? Security? The Emerging Landscape of Interpretable Agent Behavior](https://mlsanity.com/paper/1811.09722/) (2018)
- [Plan Explicability and Predictability for Robot Task Planning](https://mlsanity.com/paper/1511.08158/) (2015)
- [Towards Explainable AI Planning as a Service](https://mlsanity.com/paper/1908.05059/) (2019)

## Built on by (most cited first, 3 of 3)

- [Active Legibility in Multiagent Reinforcement Learning](https://mlsanity.com/paper/2410.20954/) (2024)
- [Symbolic Explanation of Affinity-Based Reinforcement Learning Agents with Markov Models](https://mlsanity.com/paper/2208.12627/) (2022)
- [Fog of Love: Engineering Virtuous Agent Behavior with Affinity-based Reinforcement Learning in a Game Environment](https://mlsanity.com/paper/2606.04750/) (2026)

## Abstract (arXiv)

In Reinforcement Learning interpretability generally means to provide insight into the agent's mechanisms such that its decisions are understandable by an expert upon inspection. This definition, with the resulting methods from the literature, may however fall short for online settings where the fluency of interactions prohibits deep inspections of the decision-making algorithm. To support interpretability in online settings it is useful to borrow from the Explainable Planning literature methods that focus on the legibility of the agent, by making its intention easily discernable in an observer model. As we propose in this paper, injecting legible behavior inside an agent's policy doesn't require modify components of its learning algorithm. Rather, the agent's optimal policy can be regularized for legibility by evaluating how the policy may produce observations that would make an observer infer an incorrect policy. In our formulation, the decision boundary introduced by legibility impacts the states in which the agent's policy returns an action that has high likelihood also in other policies. In these cases, a trade-off between such action, and legible/sub-optimal action is made.