mlsanityData to 7 Oct 2026

Tensor Programs IVb: Adaptive Optimization in the Infinite-Width Limit

arXiv 2308.01814 ↗First posted 3 Aug 2023cs.LG cond-mat.dis-nn cs.NEcited by 35

Not on the map. Why?

Abstract (arXiv)

Going beyond stochastic gradient descent (SGD), what new phenomena emerge in wide neural networks trained by adaptive optimizers like Adam? Here we show: The same dichotomy between feature learning and kernel behaviors (as in SGD) holds for general optimizers as well, including Adam -- albeit with a nonlinear notion of "kernel." We derive the corresponding "neural tangent" and "maximal update" limits for any architecture. Two foundational advances underlie the above results: 1) A new Tensor Program language, NEXORT, that can express how adaptive optimizers process gradients into updates. 2) The introduction of bra-ket notation to drastically simplify expressions and calculations in Tensor Programs. This work summarizes and generalizes all previous results in the Tensor Programs series of papers.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Aug 2023.

Builds on

0 of 0 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 10; 2025: 10; 2026: 12.

35 papers.

Sort:

By area:

  1. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  2. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
  3. 2025Distillation Scaling Lawscited by 59
  4. 2024The Impact of Initialization on LoRA Finetuning Dynamicscited by 53
  5. 2023Scaling Laws for Associative Memoriescited by 30
  6. 2024u-μP: The Unit-Scaled Maximal Update Parametrizationcited by 29
  7. 2024Compute Better Spent: Replacing Dense Layers with Structured Matricescited by 24
  8. 2024Super Consistency of Neural Network Landscapes and Learning Rate Transfercited by 23
  9. 2025Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Durationcited by 22
  10. 2025Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networkscited by 21
  11. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  12. 2023Leave-one-out Distinguishability in Machine Learningcited by 20
23 more
  1. 2025Hyperparameter Transfer Enables Consistent Gains of Matrix-Preconditioned Optimizers Across Scalescited by 15
  2. 2023On the Parameterization of Second-Order Optimization Effective Towards the Infinite Widthcited by 14
  3. 2024The Optimization Landscape of SGD Across the Feature Learning Strengthcited by 13
  4. 2025StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifoldcited by 13
  5. 2024Searching for Efficient Linear Layers over a Continuous Space of Structured Matricescited by 11
  6. 2025Robust Layerwise Scaling Rules by Proper Weight Decay Tuningcited by 8
  7. 2024μLO: Compute-Efficient Meta-Generalization of Learned Optimizerscited by 7
  8. 2026Deriving Hyperparameter Scaling Laws via Modern Optimization Theorycited by 7
  9. 2025Scaling Diffusion Transformers Efficiently via μPcited by 6
  10. 2025Understanding the Mechanisms of Fast Hyperparameter Transfercited by 6
  11. 2025Infinite-Width Limit of a Single Attention Layer: Analysis via Tensor Programscited by 5
  12. 2024Local Loss Optimization in the Infinite Width: Stable Parameterization of Predictive Coding Networks and Target Propagationcited by 4
  13. 2024Three Mechanisms of Feature Learning in a Linear Networkcited by 3
  14. 2026Learnable Multipliers: Freeing the Scale of Language Model Matrix Layerscited by 2
  15. 2026On the Infinite Width and Depth Limits of Predictive Coding Networkscited by 2
  16. 2026Transforming Rank: How Architecture Navigates the Spectral Pathologies of Depthcited by 2
  17. 2026SOAP, Muon, and Beyond: Pushing LLM Pretraining Scalescited by 1
  18. 2026Towards a Principled Muon under μmathsfP: Ensuring Spectral Conditions throughout Trainingcited by 0
  19. 2026How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterizationcited by 0
  20. 2026Unlocking Feature Learning in Gated Delta Networks at Scalecited by 0
  21. 2026Learning Rate Transfer for Hybrid Transformer-SSM Architecturescited by 0
  22. 2026A Deafening Silence: Catastrophic Forgetting Lives in the Output Embeddings of Tokens the Data Never Speakscited by 0
  23. 2026σTransfer: Uncertainty Transfer from Small to Large Networks under μPcited by 0