mlsanityData to 14 Oct 2026

A Meta-Learning Perspective on Transformers for Causal Language Modeling

arXiv 2310.05884 ↗First posted 9 Oct 2023cs.LG cs.AI cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

The Transformer architecture has become prominent in developing large causal language models. However, mechanisms to explain its capabilities are not well understood. Focused on the training process, here we establish a meta-learning view of the Transformer architecture when trained for the causal language modeling task, by explicating an inner optimization process within the Transformer. Further, within the inner optimization, we discover and theoretically analyze a special characteristic of the norms of learned token representations within Transformer-based causal language models. Our analysis is supported by experiments in various settings.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 36 of its 52 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

36 of 52 references.

References · 36

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2016Layer Normalizationcited by 5,669
  6. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  7. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  9. 2016Pointer Sentinel Mixture Modelscited by 3,096
  10. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  11. 2020GLU Variants Improve Transformercited by 1,804
  12. 2019Root Mean Square Layer Normalizationcited by 1,480
24 more
  1. 2022Editing Models with Task Arithmeticcited by 1,120
  2. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  3. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  4. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 796
  5. 2022Transformers learn in-context by gradient descentcited by 732
  6. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
  7. 2018Bilevel Programming for Hyperparameter Optimization and Meta-Learningcited by 405
  8. 2023Larger language models do in-context learning differentlycited by 394
  9. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
  10. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 308
  11. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  12. 2019Visualizing and Measuring the Geometry of BERTcited by 216
  13. 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
  14. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Modelscited by 154
  15. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 141
  16. 2023Uncovering mesa-optimization algorithms in Transformerscited by 100
  17. 2020BERTology Meets Biology: Interpreting Attention in Protein Language Modelscited by 85
  18. 2022Transformers as Meta-Learners for Implicit Neural Representationscited by 79
  19. 2019Meta-Curvaturecited by 70
  20. 2021Probing BERT in Hyperbolic Spacescited by 50
  21. 2022The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attentioncited by 46
  22. 2020Topic Modeling with Contextualized Word Representation Clusterscited by 41
  23. 2021Not All Models Localize Linguistic Knowledge in the Same Place: A Layer-wise Probing on BERToids' Representationscited by 16
  24. 2022Transductive Decoupled Variational Inference for Few-Shot Classificationcited by 7

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 2; 2025: 1.

5 papers.

  1. 2023Rethinking Large Language Models in Mental Health Applicationscited by 16
  2. 2024Efficient Language Adaptive Pre-training: Extending State-of-the-Art Large Language Models for Polishcited by 4
  3. 2025Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Headscited by 4
  4. 2023Semantically Grounded QFormer for Efficient Vision Language Understandingcited by 2
  5. 2024SwitchCIT: Switching for Continual Instruction Tuningcited by 1