# Shallow-Deep Networks: Understanding and Mitigating Network Overthinking

arXiv: https://arxiv.org/abs/1810.07052 · HTML: https://mlsanity.com/paper/1810.07052/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Oct 2018.

## Built on by (most cited first, 7 of 7)

- [F-PABEE: Flexible-patience-based Early Exiting for Single-label and Multi-label text Classification Tasks](https://mlsanity.com/paper/2305.11916/) (2023)
- [SHARCS: Efficient Transformers through Routing with Dynamic Width Sub-networks](https://mlsanity.com/paper/2310.12126/) (2023)
- [From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.12904/) (2026)
- [Energy-Aware Deep Learning on Resource-Constrained Hardware](https://mlsanity.com/paper/2505.12523/) (2025)
- [ACRoBat: Optimizing Auto-batching of Dynamic Deep Learning at Compile Time](https://mlsanity.com/paper/2305.10611/) (2023)
- [A Comparative Study of CNN Optimization Methods for Edge AI: Exploring the Role of Early Exits](https://mlsanity.com/paper/2604.14789/) (2026)
- [SanSi: A Looped Typed Decision Model for System 1.5 Thinking](https://mlsanity.com/paper/2610.07730/) (2026)

## Abstract (arXiv)

We characterize a prevalent weakness of deep neural networks (DNNs)---overthinking---which occurs when a DNN can reach correct predictions before its final layer. Overthinking is computationally wasteful, and it can also be destructive when, by the final layer, a correct prediction changes into a misclassification. Understanding overthinking requires studying how each prediction evolves during a DNN's forward pass, which conventionally is opaque. For prediction transparency, we propose the Shallow-Deep Network (SDN), a generic modification to off-the-shelf DNNs that introduces internal classifiers. We apply SDN to four modern architectures, trained on three image classification tasks, to characterize the overthinking problem. We show that SDNs can mitigate the wasteful effect of overthinking with confidence-based early exits, which reduce the average inference cost by more than 50% and preserve the accuracy. We also find that the destructive effect occurs for 50% of misclassifications on natural inputs and that it can be induced, adversarially, with a recent backdooring attack. To mitigate this effect, we propose a new confusion metric to quantify the internal disagreements that will likely lead to misclassifications.