# Methods and Mechanisms for Interactive Novelty Handling in Adversarial Environments

arXiv: https://arxiv.org/abs/2302.14208 · HTML: https://mlsanity.com/paper/2302.14208/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 50 references that are in the map.
- First posted on arXiv 28 Feb 2023.

## Builds on (most cited first, 21 of 21)

- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [Deep Reinforcement Learning from Self-Play in Imperfect-Information Games](https://mlsanity.com/paper/1603.01121/) (2016)
- [Multiagent Bidirectionally-Coordinated Nets: Emergence of Human-level Coordination in Learning to Play StarCraft Combat Games](https://mlsanity.com/paper/1703.10069/) (2017)
- [Solving Imperfect-Information Games via Discounted Regret Minimization](https://mlsanity.com/paper/1809.04040/) (2018)
- [Clingo = ASP + Control: Preliminary Report](https://mlsanity.com/paper/1405.3694/) (2014)
- [Bridging the Gap: Providing Post-Hoc Symbolic Explanations for Sequential Decision-Making Problems with Inscrutable Representations](https://mlsanity.com/paper/2002.01080/) (2020)
- [Depth-Limited Solving for Imperfect-Information Games](https://mlsanity.com/paper/1805.08195/) (2018)
- [SPOTTER: Extending Symbolic Planning Operators through Targeted Reinforcement Learning](https://mlsanity.com/paper/2012.13037/) (2020)
- [Variational Autoencoders for Opponent Modeling in Multi-Agent Systems](https://mlsanity.com/paper/2001.10829/) (2020)
- [Symbol Guided Hindsight Priors for Reward Learning from Human Preferences](https://mlsanity.com/paper/2210.09151/) (2022)
- [Detecting and Adapting to Novelty in Games](https://mlsanity.com/paper/2106.02204/) (2021)
- [Decision Making in Monopoly using a Hybrid Deep Reinforcement Learning Approach](https://mlsanity.com/paper/2103.00683/) (2021)
- [Opponent Learning Awareness and Modelling in Multi-Objective Normal Form Games](https://mlsanity.com/paper/2011.07290/) (2020)
- [Advice Conformance Verification by Reinforcement Learning agents for Human-in-the-Loop](https://mlsanity.com/paper/2210.03455/) (2022)
- [Towards customizable reinforcement learning agents: Enabling preference specification through online vocabulary expansion](https://mlsanity.com/paper/2210.15096/) (2022)
- [Exploiting Unlabeled Data for Feedback Efficient Human Preference based Reinforcement Learning](https://mlsanity.com/paper/2302.08738/) (2023)
- [Computing Policies That Account For The Effects Of Human Agent Uncertainty During Execution In Markov Decision Processes](https://mlsanity.com/paper/2109.07436/) (2021)
- [Data Driven Reward Initialization for Preference based Reinforcement Learning](https://mlsanity.com/paper/2302.08733/) (2023)
- [A State Augmentation based approach to Reinforcement Learning from Human Preferences](https://mlsanity.com/paper/2302.08734/) (2023)
- [Learning to Play Imperfect-Information Games by Imitating an Oracle Planner](https://mlsanity.com/paper/2012.12186/) (2020)
- [Controlling Synthetic Characters in Simulations: A Case for Cognitive Architectures and Sigma](https://mlsanity.com/paper/2101.02231/) (2021)

## Abstract (arXiv)

Learning to detect, characterize and accommodate novelties is a challenge that agents operating in open-world domains need to address to be able to guarantee satisfactory task performance. Certain novelties (e.g., changes in environment dynamics) can interfere with the performance or prevent agents from accomplishing task goals altogether. In this paper, we introduce general methods and architectural mechanisms for detecting and characterizing different types of novelties, and for building an appropriate adaptive model to accommodate them utilizing logical representations and reasoning methods. We demonstrate the effectiveness of the proposed methods in evaluations performed by a third party in the adversarial multi-agent board game Monopoly. The results show high novelty detection and accommodation rates across a variety of novelty types, including changes to the rules of the game, as well as changes to the agent's action capabilities.