mlsanityData to 7 Oct 2026

Adafactor: Adaptive Learning Rates with Sublinear Memory Cost

arXiv 1804.04235 ↗First posted 11 Apr 2018cs.LG cs.AI stat.MLcited by 896

Not on the map. Why?

Cited by 31 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In several recently proposed stochastic optimization methods (e.g. RMSProp, Adam, Adadelta), parameter updates are scaled by the inverse square roots of exponential moving averages of squared past gradients. Maintaining these per-parameter second-moment estimators requires memory equal to the number of parameters. For the case of neural network weight matrices, we propose maintaining only the per-row and per-column sums of these moving averages, and estimating the per-parameter second moments based on these sums. We demonstrate empirically that this method produces similar results to the baseline. Secondly, we show that adaptive methods can produce larger-than-desired updates when the decay rate of the second moment accumulator is too slow. We propose update clipping and a gradually increasing decay rate scheme as remedies. Combining these methods and dropping momentum, we achieve comparable results to the published Adam regime in training the Transformer model on the WMT 2014 English-German machine translation task, while using very little auxiliary storage in the optimizer. Finally, we propose scaling the parameter updates based on the scale of the parameters themselves.

Key facts. Cited by 896 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 79; 2022: 155; 2023: 214; 2024: 185; 2025: 142; 2026: 120.

896 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2023A Survey of Large Language Modelscited by 2,537
  7. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  8. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  9. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  10. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  11. 2021Scaling Vision Transformerscited by 993
  12. 2022Red Teaming Language Models with Language Modelscited by 983
38 more
  1. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  2. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  3. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  4. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 767
  5. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  6. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  7. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  8. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  9. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  10. 2022Mixture-of-Experts with Expert Choice Routingcited by 611
  11. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  12. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  13. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  14. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  15. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  16. 2021Large Dual Encoders Are Generalizable Retrieverscited by 454
  17. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  18. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  19. 2023Symbolic Discovery of Optimization Algorithmscited by 423
  20. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  21. 2021PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Modelscited by 402
  22. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  23. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  24. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  25. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  26. 2021Efficient Attentions for Long Document Summarizationcited by 342
  27. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  28. 2023Scaling Open-Vocabulary Object Detectioncited by 335
  29. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  30. 2023Synthetic Data from Diffusion Models Improves ImageNet Classificationcited by 326
  31. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  32. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  33. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  34. 2022Confident Adaptive Language Modelingcited by 297
  35. 2022UL2: Unifying Language Learning Paradigmscited by 292
  36. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  37. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  38. 2022Teaching language models to support answers with verified quotescited by 290

50 most cited of 896 All 896 (JSON)