Impact of Positional Encoding: Clean and Adversarial Rademacher Complexity for Transformers under In-Context Regression
Not on the map. Why?
Abstract (arXiv)
Positional encoding (PE) is a core architectural component of Transformers, yet its impact on the Transformer's generalization and robustness remains unclear. In this work, we provide the first generalization analysis for a single-layer Transformer under in-context regression that explicitly accounts for a completely trainable PE module. Our result shows that PE systematically enlarges the generalization gap. Extending to the adversarial setting, we derive the adversarial Rademacher generalization bound. We find that the gap between models with and without PE is magnified under attack, demonstrating that PE amplifies the vulnerability of models. Our bounds are empirically validated by a simulation study. Together, this work establishes a new framework for understanding the clean and adversarial generalization in ICL with PE.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 32 of its 56 references that are in the map. First posted on arXiv 10 Dec 2025.
Builds on
32 of 56 references.
References · 32
- 2017Attention Is All You Needcited by 60,361
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
- 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
- 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
- 2022Transformers learn in-context by gradient descentcited by 732
- 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
- 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 672
- 2023Prompt Injection attack against LLM-integrated Applicationscited by 605
- 2021MetaICL: Learning to Learn In Contextcited by 454
20 more
- 2023Trained Transformers Learn Linear Models In-Contextcited by 337
- 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 279
- 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 222
- 2022Active Example Selection for In-Context Learningcited by 196
- 2023Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attackscited by 194
- 2023Pretraining task diversity and the emergence of non-Bayesian in-context learning for regressioncited by 146
- 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
- 2018Rademacher Complexity for Adversarially Robust Generalizationcited by 132
- 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 102
- 2019VC Classes are Adversarially Robustly Learnable, but Only Improperlycited by 83
- 2022Improving In-Context Few-Shot Learning via Self-Supervised Trainingcited by 57
- 2023Attention Scheme Inspired Softmax Regressioncited by 49
- 2022Stability Analysis and Generalization Bounds of Adversarial Trainingcited by 30
- 2023Sequence Length Independent Norm-Based Generalization Bounds for Transformerscited by 29
- 2018Improved Generalization Bounds for Adversarially Robust Learningcited by 22
- 2024Data Poisoning for In-context Learningcited by 21
- 2022Adversarial Rademacher Complexity of Deep Neural Networkscited by 18
- 2024Superiority of Multi-Head Attention in In-Context Linear Regressioncited by 17
- 2024Theoretical Understanding of In-Context Learning in Shallow Transformers with Unstructured Datacited by 5
- 2025Theoretical Analysis of Positional Encodings in Transformer Models: Impact on Expressiveness and Generalizationcited by 1
Built on by
Citing papers per year: 2026: 2.
2 papers.