# Towards a Law of Iterated Expectations for Heuristic Estimators

arXiv: https://arxiv.org/abs/2410.01290 · HTML: https://mlsanity.com/paper/2410.01290/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 28 references that are in the map.
- First posted on arXiv 2 Oct 2024.

## Builds on (most cited first, 9 of 9)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [Certified Defenses against Adversarial Examples](https://mlsanity.com/paper/1801.09344/) (2018)
- [Multiaccuracy: Black-Box Post-Processing for Fairness in Classification](https://mlsanity.com/paper/1805.12317/) (2018)
- [Compact Proofs of Model Performance via Mechanistic Interpretability](https://mlsanity.com/paper/2406.11779/) (2024)
- [Logical Induction](https://mlsanity.com/paper/1609.03543/) (2016)
- [Formalizing the presumption of independence](https://mlsanity.com/paper/2211.06738/) (2022)
- [On approximability of the Permanent of PSD matrices](https://mlsanity.com/paper/2404.10959/) (2024)

## Abstract (arXiv)

Christiano et al. (2022) define a *heuristic estimator* to be a hypothetical algorithm that estimates the values of mathematical expressions from arguments. In brief, a heuristic estimator mathbbG takes as input a mathematical expression Y and a formal "heuristic argument" π, and outputs an estimate mathbbG(Y mid π) of Y. In this work, we argue for the informal principle that a heuristic estimator ought not to be able to predict its own errors, and we explore approaches to formalizing this principle. Most simply, the principle suggests that mathbbG(Y - mathbbG(Y mid π) mid π) ought to equal zero for all Y and π. We argue that an ideal heuristic estimator ought to satisfy two stronger properties in this vein, which we term *iterated estimation* (by analogy to the law of iterated expectations) and *error orthogonality*. Although iterated estimation and error orthogonality are intuitively appealing, it can be difficult to determine whether a given heuristic estimator satisfies the properties. As an alternative approach, we explore *accuracy*: a property that (roughly) states that mathbbG has zero average error over a distribution of mathematical expressions. However, in the context of two estimation problems, we demonstrate barriers to creating an accurate heuristic estimator. We finish by discussing challenges and potential paths forward for finding a heuristic estimator that accords with our intuitive understanding of how such an estimator ought to behave, as well as the potential applications of heuristic estimators to understanding the behavior of neural networks.