# Towards a theory of out-of-distribution learning

arXiv: https://arxiv.org/abs/2109.14501 · HTML: https://mlsanity.com/paper/2109.14501/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 74 references that are in the map.
- First posted on arXiv 29 Sep 2021.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [Taskonomy: Disentangling Task Transfer Learning](https://mlsanity.com/paper/1804.08328/) (2018)
- [Domain Generalization via Invariant Feature Representation](https://mlsanity.com/paper/1301.2115/) (2013)
- [Inductive Biases for Deep Learning of Higher-Level Cognition](https://mlsanity.com/paper/2011.15091/) (2020)
- [Task-Free Continual Learning](https://mlsanity.com/paper/1812.03596/) (2018)
- [The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence](https://mlsanity.com/paper/2002.06177/) (2020)
- [Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects](https://mlsanity.com/paper/1811.11553/) (2018)
- [Online Meta-Learning](https://mlsanity.com/paper/1902.08438/) (2019)
- [A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning](https://mlsanity.com/paper/2001.00689/) (2020)
- [Don't forget, there is more than forgetting: new metrics for Continual Learning](https://mlsanity.com/paper/1810.13166/) (2018)
- [Task Agnostic Continual Learning Using Online Variational Bayes](https://mlsanity.com/paper/1803.10123/) (2018)
- [Out of Distribution Generalization in Machine Learning](https://mlsanity.com/paper/2103.02667/) (2021)
- [The Unreasonable Effectiveness of Deep Learning in Artificial Intelligence](https://mlsanity.com/paper/2002.04806/) (2020)
- [Task Agnostic Continual Learning via Meta Learning](https://mlsanity.com/paper/1906.05201/) (2019)

## Built on by (most cited first, 11 of 11)

- [Dimensionless machine learning: Imposing exact units equivariance](https://mlsanity.com/paper/2204.00887/) (2022)
- [Tracking the perspectives of interacting language models](https://mlsanity.com/paper/2406.11938/) (2024)
- [A Domain-Agnostic Approach for Characterization of Lifelong Learning Systems](https://mlsanity.com/paper/2301.07799/) (2023)
- [Mental State Classification Using Multi-graph Features](https://mlsanity.com/paper/2203.00516/) (2022)
- [Statistical Context Detection for Deep Lifelong Reinforcement Learning](https://mlsanity.com/paper/2405.19047/) (2024)
- [Prospective Learning: Principled Extrapolation to the Future](https://mlsanity.com/paper/2201.07372/) (2022)
- [Extremely Simple Streaming Forest](https://mlsanity.com/paper/2110.08483/) (2021)
- [A benchmark with decomposed distribution shifts for 360 monocular depth estimation](https://mlsanity.com/paper/2112.00432/) (2021)
- [Deep and shallow data science for multi-scale optical neuroscience](https://mlsanity.com/paper/2402.08811/) (2024)
- [Disentangled Representations for Causal Cognition](https://mlsanity.com/paper/2407.00744/) (2024)
- [Learning Model Successors](https://mlsanity.com/paper/2502.00197/) (2025)

## Abstract (arXiv)

Learning is a process wherein a learning agent enhances its performance through exposure of experience or data. Throughout this journey, the agent may encounter diverse learning environments. For example, data may be presented to the leaner all at once, in multiple batches, or sequentially. Furthermore, the distribution of each data sample could be either identical and independent (iid) or non-iid. Additionally, there may exist computational and space constraints for the deployment of the learning algorithms. The complexity of a learning task can vary significantly, depending on the learning setup and the constraints imposed upon it. However, it is worth noting that the current literature lacks formal definitions for many of the in-distribution and out-of-distribution learning paradigms. Establishing proper and universally agreed-upon definitions for these learning setups is essential for thoroughly exploring the evolution of ideas across different learning scenarios and deriving generalized mathematical bounds for these learners. In this paper, we aim to address this issue by proposing a chronological approach to defining different learning tasks using the provably approximately correct (PAC) learning framework. We will start with in-distribution learning and progress to recently proposed lifelong or continual learning. We employ consistent terminology and notation to demonstrate how each of these learning frameworks represents a specific instance of a broader, more generalized concept of learnability. Our hope is that this work will inspire a universally agreed-upon approach to quantifying different types of learning, fostering greater understanding and progress in the field.