# Anticipatory Thinking Challenges in Open Worlds: Risk Management

arXiv: https://arxiv.org/abs/2306.13157 · HTML: https://mlsanity.com/paper/2306.13157/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 9 references that are in the map.
- First posted on arXiv 22 Jun 2023.

## Builds on (most cited first, 5 of 5)

- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images](https://mlsanity.com/paper/1412.1897/) (2014)
- [The NetHack Learning Environment](https://mlsanity.com/paper/2006.13760/) (2020)
- [Generating Natural Adversarial Examples](https://mlsanity.com/paper/1710.11342/) (2017)

## Built on by (most cited first, 1 of 1)

- [What's the next frontier for Data-centric AI? Data Savvy Agents](https://mlsanity.com/paper/2511.01015/) (2025)

## Abstract (arXiv)

Anticipatory thinking drives our ability to manage risk - identification and mitigation - in everyday life, from bringing an umbrella when it might rain to buying car insurance. As AI systems become part of everyday life, they too have begun to manage risk. Autonomous vehicles log millions of miles, StarCraft and Go agents have similar capabilities to humans, implicitly managing risks presented by their opponents. To further increase performance in these tasks, out-of-distribution evaluation can characterize a model's bias, what we view as a type of risk management. However, learning to identify and mitigate low-frequency, high-impact risks is at odds with the observational bias required to train machine learning models. StarCraft and Go are closed-world domains whose risks are known and mitigations well documented, ideal for learning through repetition. Adversarial filtering datasets provide difficult examples but are laborious to curate and static, both barriers to real-world risk management. Adversarial robustness focuses on model poisoning under the assumption there is an adversary with malicious intent, without considering naturally occurring adversarial examples. These methods are all important steps towards improving risk management but do so without considering open-worlds. We unify these open-world risk management challenges with two contributions. The first is our perception challenges, designed for agents with imperfect perceptions of their environment whose consequences have a high impact. Our second contribution are cognition challenges, designed for agents that must dynamically adjust their risk exposure as they identify new risks and learn new mitigations. Our goal with these challenges is to spur research into solutions that assess and improve the anticipatory thinking required by AI agents to manage risk in open-worlds and ultimately the real-world.