mlsanityData to 7 Oct 2026

PonderNet: Learning to Ponder

arXiv 2107.05407 ↗First posted 12 Jul 2021cs.LG cs.AI cs.CCcited by 156

Not on the map. Why?

Cited by 37 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In standard neural networks the amount of computation used grows with the size of the inputs, but not with the complexity of the problem being learnt. To overcome this limitation we introduce PonderNet, a new algorithm that learns to adapt the amount of computation based on the complexity of the problem at hand. PonderNet learns end-to-end the number of computational steps to achieve an effective compromise between training prediction accuracy, computational cost and generalization. On a complex synthetic problem, PonderNet dramatically improves performance over previous adaptive computation methods and additionally succeeds at extrapolation tests where traditional neural networks fail. Also, our method matched the current state of the art results on a real world question and answering dataset, but using less compute. Finally, PonderNet reached state of the art results on a complex task designed to test the reasoning capabilities of neural networks.1

Key facts. Cited by 156 papers in the map (data through 7 Oct 2026). Builds on 12 of its 17 references that are in the map. First posted on arXiv 12 Jul 2021.

Builds on

12 of 17 references.

References · 12

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  5. 2016Adaptive Computation Time for Recurrent Neural Networkscited by 479
  6. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  7. 2017Adaptive Neural Networks for Efficient Inferencecited by 161
  8. 2019Neural Execution of Graph Algorithmscited by 118
  9. 2016Hierarchical Multiscale Recurrent Neural Networkscited by 102
  10. 2017Skip RNN: Learning to Skip State Updates in Recurrent Neural Networkscited by 49
  11. 2020MEMO: A Deep Network for Flexible Combination of Episodic Memoriescited by 21
  12. 2017Learning to Skim Textcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 17; 2023: 14; 2024: 11; 2025: 24; 2026: 82.

156 papers.

Sort:

By area:

  1. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  2. 2022Galactica: A Large Language Model for Sciencecited by 736
  3. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
  4. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  5. 2025Hierarchical Reasoning Modelcited by 167
  6. 2025Scaling Latent Reasoning via Looped Language Modelscited by 145
  7. 2022Faithful Reasoning Using Large Language Modelscited by 130
  8. 2022The CLRS Algorithmic Reasoning Benchmarkcited by 103
  9. 2024Learning How Hard to Think: Input-Adaptive Allocation of LM Computationcited by 67
  10. 2021The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalizationcited by 62
  11. 2025Do Language Models Use Their Depth Efficiently?cited by 48
  12. 2024Learning Iterative Reasoning through Energy Diffusioncited by 36
38 more
  1. 2022End-to-end Algorithm Synthesis with Recurrent Networks: Logical Extrapolation Without Overthinkingcited by 33
  2. 2022Path Independent Equilibrium Models Can Better Exploit Test-Time Computationcited by 33
  3. 2022Learning Iterative Reasoning through Energy Minimizationcited by 32
  4. 2023Learning to Reason and Memorize with Self-Notescited by 30
  5. 2025Large Language Models Are Human-Like Internallycited by 28
  6. 2026A Mechanistic Analysis of Looped Reasoning Language Modelscited by 27
  7. 2024Mixture of Nested Experts: Adaptive Processing of Visual Tokenscited by 25
  8. 2023Adaptive Computation with Elastic Input Sequencecited by 22
  9. 2025Implicit Language Models are RNNs: Balancing Parallelization and Expressivitycited by 21
  10. 2025Continuous Thought Machinescited by 21
  11. 2026Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformerscited by 21
  12. 2024Rational Metareasoning for Large Language Modelscited by 20
  13. 2022Dynamic Split Computing for Efficient Deep Edge Intelligencecited by 18
  14. 2023Can neural networks do arithmetic? A survey on the elementary numerical skills of state-of-the-art deep learning modelscited by 18
  15. 2023ModuleFormer: Modularity Emerges from Mixture-of-Expertscited by 17
  16. 2022Unsupervised Learning of Temporal Abstractions with Slot-based Transformerscited by 14
  17. 2024Limits of Transformer Language Models on Learning to Compose Algorithmscited by 13
  18. 2024Scaling Stick-Breaking Attention: An Efficient Implementation and In-depth Studycited by 13
  19. 2024CEEBERT: Cross-Domain Inference in Early Exit BERTcited by 12
  20. 2025Padding Tone: A Mechanistic Analysis of Padding Tokens in T2I Modelscited by 12
  21. 2026Training-Free Looped Transformerscited by 12
  22. 2026Adaptive Loops and Memory in Transformers: Think Harder or Know More?cited by 11
  23. 2021Recurrent Vision Transformer for Solving Visual Reasoning Problemscited by 9
  24. 2022Efficient High-Resolution Deep Learning: A Surveycited by 9
  25. 2023Monotonic Location Attention for Length Generalizationcited by 9
  26. 2026AdaPonderLM: Gated Pondering Language Models with Token-Wise Adaptive Depthcited by 9
  27. 2026Stability and Generalization in Looped Transformerscited by 9
  28. 2021Dyna-bAbI: unlocking bAbI's potential with dynamic synthetic benchmarkingcited by 7
  29. 2022Transformers discover an elementary calculation system exploiting local attention and grid-like problem representationcited by 7
  30. 2023Adaptivity and Modularity for Efficient Generalization Over Task Complexitycited by 7
  31. 2025LaRe: Latent Refocusing for Multimodal Reasoningcited by 7
  32. 2026Adaptive Depth in Looped Transformers: Diagnosing Learned Halting Gates and Trajectory Readoutscited by 7
  33. 2021Iterative Decoding for Compositional Generalization in Transformerscited by 6
  34. 2022PALBERT: Teaching ALBERT to Pondercited by 6
  35. 2022Explainability Via Causal Self-Talkcited by 6
  36. 2023Adaptive recurrent vision performs zero-shot computation scaling to unseen difficulty levelscited by 6
  37. 2025BEEM: Boosting Performance of Early Exit DNNs using Multi-Exit Classifiers as Expertscited by 6
  38. 2026Understanding Dynamic Compute Allocation in Recurrent Transformerscited by 6

50 most cited of 156 All 156 (JSON)