# Incorporating Behavioral Constraints in Online AI Systems

arXiv: https://arxiv.org/abs/1809.05720 · HTML: https://mlsanity.com/paper/1809.05720/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Sep 2018.

## Built on by (most cited first, 16 of 16)

- [Thinking Fast and Slow in AI: the Role of Metacognition](https://mlsanity.com/paper/2110.01834/) (2021)
- [Making Human-Like Trade-offs in Constrained Environments by Learning from Demonstrations](https://mlsanity.com/paper/2109.11018/) (2021)
- [Maximum Causal Entropy Inverse Constrained Reinforcement Learning](https://mlsanity.com/paper/2305.02857/) (2023)
- [Survey: Multi-Armed Bandits Meet Large Language Models](https://mlsanity.com/paper/2505.13355/) (2025)
- [Optimal Epidemic Control as a Contextual Combinatorial Bandit with Budget](https://mlsanity.com/paper/2106.15808/) (2021)
- [When Is It Acceptable to Break the Rules? Knowledge Representation of Moral Judgement Based on Empirical Data](https://mlsanity.com/paper/2201.07763/) (2022)
- [Survey on Applications of Neurosymbolic Artificial Intelligence](https://mlsanity.com/paper/2209.12618/) (2022)
- [Reinforcement Learning Guided by Provable Normative Compliance](https://mlsanity.com/paper/2203.16275/) (2022)
- [Reinforcement Learning and Machine ethics:a systematic review](https://mlsanity.com/paper/2407.02425/) (2024)
- [Learning Behavioral Soft Constraints from Demonstrations](https://mlsanity.com/paper/2202.10407/) (2022)
- [Advising Autonomous Cars about the Rules of the Road](https://mlsanity.com/paper/2209.14035/) (2022)
- [A Reinforcement Learning Framework for Online Speaker Diarization](https://mlsanity.com/paper/2302.10924/) (2023)
- [Learning the Value Systems of Agents with Preference-based and Inverse Reinforcement Learning](https://mlsanity.com/paper/2602.04518/) (2026)
- [Etat de l'art sur l'application des bandits multi-bras](https://mlsanity.com/paper/2101.00001/) (2021)
- [HAVA: Hybrid Approach to Value-Alignment through Reward Weighing for Reinforcement Learning](https://mlsanity.com/paper/2505.15011/) (2025)
- [Learning the Value Systems of Societies with Preference-based Multi-objective Reinforcement Learning](https://mlsanity.com/paper/2602.08835/) (2026)

## Abstract (arXiv)

AI systems that learn through reward feedback about the actions they take are increasingly deployed in domains that have significant impact on our daily life. However, in many cases the online rewards should not be the only guiding criteria, as there are additional constraints and/or priorities imposed by regulations, values, preferences, or ethical principles. We detail a novel online agent that learns a set of behavioral constraints by observation and uses these learned constraints as a guide when making decisions in an online setting while still being reactive to reward feedback. To define this agent, we propose to adopt a novel extension to the classical contextual multi-armed bandit setting and we provide a new algorithm called Behavior Constrained Thompson Sampling (BCTS) that allows for online learning while obeying exogenous constraints. Our agent learns a constrained policy that implements the observed behavioral constraints demonstrated by a teacher agent, and then uses this constrained policy to guide the reward-based online exploration and exploitation. We characterize the upper bound on the expected regret of the contextual bandit algorithm that underlies our agent and provide a case study with real world data in two application domains. Our experiments show that the designed agent is able to act within the set of behavior constraints without significantly degrading its overall reward performance.