mlsanityData to 14 Oct 2026

The Garden of Forking Paths: Observing Dynamic Parameters Distribution in Large Language Models

arXiv 2403.08739 ↗First posted 13 Mar 2024cs.CL cond-mat.dis-nn cond-mat.stat-mechcited by 1

Not on the map. Why?

Abstract (arXiv)

A substantial gap persists in understanding the reasons behind the exceptional performance of the Transformer architecture in NLP. A particularly unexplored area involves the mechanistic description of how the distribution of parameters evolves over time during training. In this work we suggest that looking at the time evolution of the statistic distribution of model parameters, and specifically at bifurcation effects, can help understanding the model quality, potentially reducing training costs and evaluation efforts and empirically showing the reasons behind the effectiveness of weights sparsification.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 41 of its 55 references that are in the map. First posted on arXiv 13 Mar 2024.

Builds on

41 of 55 references.

References · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2016Layer Normalizationcited by 5,669
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  9. 2022Emergent Abilities of Large Language Modelscited by 2,512
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
29 more
  1. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  4. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  5. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  6. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  7. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
  8. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 713
  9. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  10. 2020Transformer Interpretability Beyond Attention Visualizationcited by 627
  11. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  12. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 523
  13. 2023Language Models Represent Space and Timecited by 316
  14. 2024The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bitscited by 308
  15. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  16. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  17. 2022Towards Understanding Grokking: An Effective Theory of Representation Learningcited by 230
  18. 2023A mathematical perspective on Transformerscited by 164
  19. 2023Birth of a Transformer: A Memory Viewpointcited by 161
  20. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 156
  21. 2022Omnigrok: Grokking Beyond Algorithmic Datacited by 146
  22. 2023Are Emergent Abilities in Large Language Models just In-Context Learning?cited by 109
  23. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  24. 2023A Tale of Two Circuits: Grokking as Competition of Sparse and Dense Subnetworkscited by 83
  25. 2017Critical Learning Periods in Deep Neural Networkscited by 79
  26. 2023Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Modelscited by 67
  27. 2023Seeing is Believing: Brain-Inspired Modular Training for Mechanistic Interpretabilitycited by 38
  28. 2015Crowd Behavior Analysis: A Review where Physics meets Biologycited by 5
  29. 2024Evaluating Brain-Inspired Modular Training in Automated Circuit Discovery for Mechanistic Interpretabilitycited by 5

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Enlightenment Period Improving DNN Performancecited by 0