# Orthogonalized SGD and Nested Architectures for Anytime Neural Networks

arXiv: https://arxiv.org/abs/2008.06635 · HTML: https://mlsanity.com/paper/2008.06635/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Aug 2020.

## Built on by (most cited first, 5 of 5)

- [Mixture of Nested Experts: Adaptive Processing of Visual Tokens](https://mlsanity.com/paper/2407.19985/) (2024)
- [TangoBERT: Reducing Inference Cost by using Cascaded Architecture](https://mlsanity.com/paper/2204.06271/) (2022)
- [Accelerated Training via Incrementally Growing Neural Networks using Variance Transfer and Learning Rate Adaptation](https://mlsanity.com/paper/2306.12700/) (2023)
- [Adaptive Depth Networks with Skippable Sub-Paths](https://mlsanity.com/paper/2312.16392/) (2023)
- [Comparing and Combining Approximate Computing Frameworks](https://mlsanity.com/paper/2102.08771/) (2021)

## Abstract (arXiv)

We propose a novel variant of SGD customized for training network architectures that support anytime behavior: such networks produce a series of increasingly accurate outputs over time. Efficient architectural designs for these networks focus on re-using internal state; subnetworks must produce representations relevant for both immediate prediction as well as refinement by subsequent network stages. We consider traditional branched networks as well as a new class of recursively nested networks. Our new optimizer, Orthogonalized SGD, dynamically re-balances task-specific gradients when training a multitask network. In the context of anytime architectures, this optimizer projects gradients from later outputs onto a parameter subspace that does not interfere with those from earlier outputs. Experiments demonstrate that training with Orthogonalized SGD significantly improves generalization accuracy of anytime networks.