# Good and safe uses of AI Oracles

arXiv: https://arxiv.org/abs/1711.05541 · HTML: https://mlsanity.com/paper/1711.05541/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Nov 2017.

## Built on by (most cited first, 14 of 14)

- [Alignment of Language Agents](https://mlsanity.com/paper/2103.14659/) (2021)
- [Goal Misgeneralization in Deep Reinforcement Learning](https://mlsanity.com/paper/2105.14111/) (2021)
- [Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems](https://mlsanity.com/paper/2405.06624/) (2024)
- [Evaluating Superhuman Models with Consistency Checks](https://mlsanity.com/paper/2306.09983/) (2023)
- [General agents contain world models](https://mlsanity.com/paper/2506.01622/) (2025)
- [Conditioning Predictive Models: Risks and Strategies](https://mlsanity.com/paper/2302.00805/) (2023)
- [Incentivizing honest performative predictions with proper scoring rules](https://mlsanity.com/paper/2305.17601/) (2023)
- [From AGI to ASI](https://mlsanity.com/paper/2606.12683/) (2026)
- [Scoring Rules for Performative Binary Prediction](https://mlsanity.com/paper/2207.02847/) (2022)
- [Negative Human Rights as a Basis for Long-term AI Safety and Regulation](https://mlsanity.com/paper/2208.14788/) (2022)
- [Counterfactual Planning in AGI Systems](https://mlsanity.com/paper/2102.00834/) (2021)
- [Joint Scoring Rules: Zero-Sum Competition Avoids Performative Prediction](https://mlsanity.com/paper/2412.20732/) (2024)
- [Statistical Scenario Modelling and Lookalike Distributions for Multi-Variate AI Risk](https://mlsanity.com/paper/2502.14491/) (2025)
- [Unbiased Canonical Set-Valued Oracles Via Lattice Theory](https://mlsanity.com/paper/2606.26418/) (2026)

## Abstract (arXiv)

It is possible that powerful and potentially dangerous artificial intelligence (AI) might be developed in the future. An Oracle is a design which aims to restrain the impact of a potentially dangerous AI by restricting the agent to no actions besides answering questions. Unfortunately, most Oracles will be motivated to gain more control over the world by manipulating users through the content of their answers, and Oracles of potentially high intelligence might be very successful at this citepDBLP:journals/corr/AlfonsecaCACAR16. In this paper we present two designs for Oracles which, even under pessimistic assumptions, will not manipulate their users into releasing them and yet will still be incentivised to provide their users with helpful answers. The first design is the counterfactual Oracle -- which choses its answer as if it expected nobody to ever read it. The second design is the low-bandwidth Oracle -- which is limited by the quantity of information it can transmit.