# Loop Neural Networks for Parameter Sharing

arXiv: https://arxiv.org/abs/2409.14199 · HTML: https://mlsanity.com/paper/2409.14199/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 7 references that are in the map.
- First posted on arXiv 21 Sep 2024.

## Builds on (most cited first, 4 of 4)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [Adaptive Computation Time for Recurrent Neural Networks](https://mlsanity.com/paper/1603.08983/) (2016)
- [Depth-Adaptive Transformer](https://mlsanity.com/paper/1910.10073/) (2019)

## Built on by (most cited first, 11 of 11)

- [Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation](https://mlsanity.com/paper/2507.10524/) (2025)
- [Inner Thinking Transformer: Leveraging Dynamic Depth Scaling to Foster Adaptive Internal Thinking](https://mlsanity.com/paper/2502.13842/) (2025)
- [Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence](https://mlsanity.com/paper/2511.07384/) (2025)
- [Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers](https://mlsanity.com/paper/2606.31779/) (2026)
- [On the Residual Scaling of Looped Transformers: Stability and Transferability](https://mlsanity.com/paper/2606.18524/) (2026)
- [MeSH: Memory-as-State-Highways for Recursive Transformers](https://mlsanity.com/paper/2510.07739/) (2025)
- [Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformation](https://mlsanity.com/paper/2603.04971/) (2026)
- [Accelerating Training Speed of Tiny Recursive Models with Curriculum Guided Adaptive Recursion](https://mlsanity.com/paper/2511.08653/) (2025)
- [Sparse Growing Transformer: Training-Time Sparse Depth Allocation via Progressive Attention Looping](https://mlsanity.com/paper/2603.23998/) (2026)
- [ITS-Mina: A Harris Hawks Optimization-Based All-MLP Framework with Iterative Refinement and External Attention for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2604.27981/) (2026)
- [Statistical Gains from Looped Estimation under Parameter Budgets](https://mlsanity.com/paper/2609.25778/) (2026)

## Abstract (arXiv)

The success of large-scale language models like GPT can be attributed to their ability to efficiently predict the next token in a sequence. However, these models rely on constant computational effort regardless of the complexity of the token they are predicting, lacking the capacity for iterative refinement. In this paper, we introduce a novel Loop Neural Network, which achieves better performance by utilizing longer computational time without increasing the model size. Our approach revisits the input multiple times, refining the prediction by iteratively looping over a subset of the model with residual connections. We demonstrate the effectiveness of this method through experiments comparing versions of GPT-2 with our loop models, showing improved performance in language modeling tasks while maintaining similar parameter counts. Importantly, these improvements are achieved without the need for extra training data.