# Predicting on the Edge: Identifying Where a Larger Model Does Better

arXiv: https://arxiv.org/abs/2202.07652 · HTML: https://mlsanity.com/paper/2202.07652/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 48 references that are in the map.
- First posted on arXiv 15 Feb 2022.

## Builds on (most cited first, 30 of 31)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning](https://mlsanity.com/paper/1602.07261/) (2016)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Underspecification Presents Challenges for Credibility in Modern Machine Learning](https://mlsanity.com/paper/2011.03395/) (2020)
- [Calibration of Pre-trained Transformers](https://mlsanity.com/paper/2003.07892/) (2020)
- [An Investigation of Why Overparameterization Exacerbates Spurious Correlations](https://mlsanity.com/paper/2005.04345/) (2020)
- [Pretrained Transformers Improve Out-of-Distribution Robustness](https://mlsanity.com/paper/2004.06100/) (2020)
- [BERT Loses Patience: Fast and Robust Inference with Early Exit](https://mlsanity.com/paper/2006.04152/) (2020)
- [Professor Forcing: A New Algorithm for Training Recurrent Networks](https://mlsanity.com/paper/1610.09038/) (2016)
- [Ex Machina: Personal Attacks Seen at Scale](https://mlsanity.com/paper/1610.08914/) (2016)
- [Deep Learning Through the Lens of Example Difficulty](https://mlsanity.com/paper/2106.09647/) (2021)
- [Optimization with Non-Differentiable Constraints with Applications to Fairness, Recall, Churn, and Other Goals](https://mlsanity.com/paper/1809.04198/) (2018)
- [Characterizing Structural Regularities of Labeled Data in Overparameterized Models](https://mlsanity.com/paper/2002.03206/) (2020)
- [The MultiBERTs: BERT Reproductions for Robustness Analysis](https://mlsanity.com/paper/2106.16163/) (2021)
- [Active Testing: Sample-Efficient Model Evaluation](https://mlsanity.com/paper/2103.05331/) (2021)
- [IDK Cascades: Fast Deep Learning by Learning not to Overthink](https://mlsanity.com/paper/1706.00885/) (2017)
- [Supervised Transfer Learning at Scale for Medical Imaging](https://mlsanity.com/paper/2101.05913/) (2021)
- [Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level](https://mlsanity.com/paper/2105.06020/) (2021)
- [On the Reproducibility of Neural Network Predictions](https://mlsanity.com/paper/2102.03349/) (2021)
- [Churn Reduction via Distillation](https://mlsanity.com/paper/2106.02654/) (2021)
- [Adaptive Classification for Prediction Under a Budget](https://mlsanity.com/paper/1705.10194/) (2017)
- [When in Doubt, Summon the Titans: Efficient Inference with Large Models](https://mlsanity.com/paper/2110.10305/) (2021)

## Built on by (most cited first, 6 of 6)

- [When Does Confidence-Based Cascade Deferral Suffice?](https://mlsanity.com/paper/2307.02764/) (2023)
- [ERSAM: Neural Architecture Search For Energy-Efficient and Real-Time Social Ambiance Measurement](https://mlsanity.com/paper/2303.10727/) (2023)
- [NetDistiller: Empowering Tiny Deep Learning via In-Situ Distillation](https://mlsanity.com/paper/2310.19820/) (2023)
- [Streaming Lossless Volumetric Compression of Medical Images Using Gated Recurrent Convolutional Neural Network](https://mlsanity.com/paper/2311.16200/) (2023)
- [Dual-Model Distillation for Efficient Action Classification with Hybrid Edge-Cloud Solution](https://mlsanity.com/paper/2410.12165/) (2024)
- [Inducing Artificial Uncertainty in Language Models](https://mlsanity.com/paper/2605.13595/) (2026)

## Abstract (arXiv)

Much effort has been devoted to making large and more accurate models, but relatively little has been put into understanding which examples are benefiting from the added complexity. In this paper, we demonstrate and analyze the surprisingly tight link between a model's predictive uncertainty on individual examples and the likelihood that larger models will improve prediction on them. Through extensive numerical studies on the T5 encoder-decoder architecture, we show that large models have the largest improvement on examples where the small model is most uncertain. On more certain examples, even those where the small model is not particularly accurate, large models are often unable to improve at all, and can even perform worse than the smaller model. Based on these findings, we show that a switcher model which defers examples to a larger model when a small model is uncertain can achieve striking improvements in performance and resource usage. We also explore committee-based uncertainty metrics that can be more effective but less practical.