mlsanityData to 7 Oct 2026

Adaptive Computation Time for Recurrent Neural Networks

arXiv 1603.08983 ↗First posted 29 Mar 2016cs.NEcited by 479

Not on the map. Why?

Cited by 97 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper introduces Adaptive Computation Time (ACT), an algorithm that allows recurrent neural networks to learn how many computational steps to take between receiving an input and emitting an output. ACT requires minimal changes to the network architecture, is deterministic and differentiable, and does not add any noise to the parameter gradients. Experimental results are provided for four synthetic problems: determining the parity of binary vectors, applying binary logic operations, adding integers, and sorting real numbers. Overall, performance is dramatically improved by the use of ACT, which successfully adapts the number of computational steps to the requirements of the problem. We also present character-level language modelling results on the Hutter prize Wikipedia dataset. In this case ACT does not yield large gains in performance; however it does provide intriguing insight into the structure of the data, with more computation allocated to harder-to-predict transitions, such as spaces between words and ends of sentences. This suggests that ACT or other adaptive computation methods could provide a generic method for inferring segment boundaries in sequence data.

Key facts. Cited by 479 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Mar 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 50; 2022: 40; 2023: 50; 2024: 45; 2025: 77; 2026: 217.

479 papers.

Sort:

By area:

  1. 2022Emergent Abilities of Large Language Modelscited by 2,483
  2. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  3. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  4. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  5. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  6. 2022Galactica: A Large Language Model for Sciencecited by 736
  7. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  8. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
  9. 2021Dynamic Neural Networks: A Surveycited by 405
  10. 2021A Survey of Transformerscited by 399
  11. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  12. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 360
38 more
  1. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  2. 2022Confident Adaptive Language Modelingcited by 297
  3. 2022Transformers Learn Shortcuts to Automatacited by 260
  4. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  5. 2021GRAND: Graph Neural Diffusioncited by 258
  6. 2023Think before you speak: Training Language Models With Pause Tokenscited by 254
  7. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  8. 2022Neural Networks and the Chomsky Hierarchycited by 214
  9. 2022Scalable Adaptive Computation for Iterative Generationcited by 190
  10. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memorycited by 170
  11. 2025Hierarchical Reasoning Modelcited by 167
  12. 2021PonderNet: Learning to Pondercited by 156
  13. 2022Faithful Reasoning Using Large Language Modelscited by 130
  14. 2021IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformerscited by 127
  15. 2025Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computationcited by 120
  16. 2021The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformerscited by 117
  17. 2021Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networkscited by 108
  18. 2021MOOD: Multi-level Out-of-distribution Detectioncited by 94
  19. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  20. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  21. 2021Lessons on Parameter Sharing across Layers in Transformerscited by 85
  22. 2023Fast and Robust Early-Exiting Framework for Autoregressive Language Models with Synchronized Parallel Decodingcited by 80
  23. 2022Efficient Transformers with Dynamic Token Poolingcited by 75
  24. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  25. 2021Consistent Accelerated Inference via Confident Adaptive Transformerscited by 73
  26. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  27. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  28. 2025DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Productscited by 69
  29. 2024Learning How Hard to Think: Input-Adaptive Allocation of LM Computationcited by 67
  30. 2021The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalizationcited by 62
  31. 2021Linear algebra with transformerscited by 62
  32. 2023Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformerscited by 61
  33. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruningcited by 56
  34. 2024MoEUT: Mixture-of-Experts Universal Transformerscited by 55
  35. 2022Dynamic Multimodal Fusioncited by 54
  36. 2021A Survey on Green Deep Learningcited by 52
  37. 2021Attention, please! A survey of Neural Attention Models in Deep Learningcited by 50
  38. 2023Revisiting Token Pruning for Object Detection and Instance Segmentationcited by 50

50 most cited of 479 All 479 (JSON)