mlsanityData to 14 Oct 2026

Vocabulary In-Context Learning in Transformers: Benefits of Positional Encoding

arXiv 2511.06376 ↗First posted 9 Nov 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Numerous studies have demonstrated that the Transformer architecture possesses the capability for in-context learning (ICL). In scenarios involving function approximation, context can serve as a control parameter for the model, endowing it with the universal approximation property (UAP). In practice, context is represented by tokens from a finite set, referred to as a vocabulary, which is the case considered in this paper, i.e., vocabulary in-context learning (VICL). We demonstrate that VICL in single-layer Transformers, without positional encoding, does not possess the UAP; however, it is possible to achieve the UAP when positional encoding is included. Several sufficient conditions for the positional encoding are provided. Our findings reveal the benefits of positional encoding from an approximation theory perspective in the context of ICL.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 45 of its 60 references that are in the map. First posted on arXiv 9 Nov 2025.

Builds on

45 of 60 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  12. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
33 more
  1. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  4. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  6. 2018Self-Attention with Relative Position Representationscited by 1,120
  7. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  8. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  9. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  10. 2021Editing Factual Knowledge in Language Modelscited by 553
  11. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 406
  12. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 320
  13. 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
  14. 2022Is GPT-3 a Good Data Annotator?cited by 208
  15. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  16. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  17. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  18. 2023Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learningcited by 149
  19. 2020Minimum Width for Universal Approximationcited by 94
  20. 2023Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Contextcited by 63
  21. 2020Learning to Encode Position for Transformer with Continuous Dynamical Modelcited by 61
  22. 2022Your Transformer May Not be as Powerful as You Expectcited by 55
  23. 2021On the Expressive Power of Self-Attention Matricescited by 47
  24. 2023When Do Prompting and Prefix-Tuning Work? A Theory of Capabilities and Limitationscited by 43
  25. 2023On the Optimization and Generalization of Multi-head Attentioncited by 37
  26. 2025LLM Pretraining with Continuous Conceptscited by 36
  27. 2023Sumformer: Universal Approximation for Efficient Transformerscited by 30
  28. 2022Achieve the Minimum Width of Neural Networks for Universal Approximationcited by 26
  29. 2024Prompting a Pretrained Transformer Can Be a Universal Approximatorcited by 25
  30. 2023Exploring In-Context Learning Capabilities of Foundation Models for Generating Knowledge Graphs from Textcited by 19
  31. 2024Verbalized Machine Learning: Revisiting Machine Learning with Language Modelscited by 17
  32. 2023Minimum Width of Leaky-ReLU Neural Networks for Uniform Universal Approximationcited by 13
  33. 2023Vocabulary for Universal Approximation: A Linguistic Perspective of Mapping Compositionscited by 6

Built on by

Not cited yet.