mlsanityData to 14 Oct 2026

A Theoretical Framework for OOD Robustness in Transformers using Gevrey Classes

arXiv 2504.12991 ↗First posted 17 Apr 2025cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

We study the robustness of Transformer language models under semantic out-of-distribution (OOD) shifts, where training and test data lie in disjoint latent spaces. Using Wasserstein-1 distance and Gevrey-class smoothness, we derive sub-exponential upper bounds on prediction error. Our theoretical framework explains how smoothness governs generalization under distributional drift. We validate these findings through controlled experiments on arithmetic and Chain-of-Thought tasks with latent permutations and scalings. Results show empirical degradation aligns with our bounds, highlighting the geometric and functional principles underlying OOD generalization in Transformers.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 39 of its 50 references that are in the map. First posted on arXiv 17 Apr 2025.

Builds on

39 of 50 references.

References · 39

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  6. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  7. 2019Invariant Risk Minimizationcited by 1,854
  8. 2019Theoretically Principled Trade-off between Robustness and Accuracycited by 1,517
  9. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,485
  10. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  11. 2020Energy-based Out-of-distribution Detectioncited by 1,299
  12. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
27 more
  1. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
  2. 2018Generalizing to Unseen Domains via Adversarial Data Augmentationcited by 451
  3. 2017Joint Distribution Optimal Transportation for Domain Adaptationcited by 282
  4. 2017Zero-Shot Learning -- The Good, the Bad and the Uglycited by 255
  5. 2016Diachronic Word Embeddings Reveal Statistical Laws of Semantic Changecited by 221
  6. 2015Learning with a Wasserstein Losscited by 196
  7. 2017Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulationcited by 185
  8. 2023What and How does In-Context Learning Learn? Bayesian Model Averaging, Parameterization, and Generalizationcited by 118
  9. 2016Theoretical Analysis of Domain Adaptation with Optimal Transportcited by 105
  10. 2023Revisiting Out-of-distribution Robustness in NLP: Benchmark, Analysis, and LLMs Evaluationscited by 97
  11. 2023Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languagescited by 97
  12. 2022GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-distribution Generalization Perspectivecited by 72
  13. 2021Universal Representation Learning from Multiple Domains for Few-shot Classificationcited by 56
  14. 2024Transformers Provably Solve Parity Efficiently with Chain of Thoughtcited by 50
  15. 2021LIME: Learning Inductive Bias for Primitives of Mathematical Reasoningcited by 46
  16. 2024Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methodscited by 29
  17. 2024Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structurecited by 23
  18. 2024Can In-context Learning Really Generalize to Out-of-distribution Tasks?cited by 23
  19. 2024Training Nonlinear Transformers for Chain-of-Thought Inference: A Theoretical Generalization Analysiscited by 14
  20. 2024A Theory for Length Generalization in Learning to Reasoncited by 11
  21. 2022Evaluating Interpolation and Extrapolation Performance of Neural Retrieval Modelscited by 9
  22. 2025Beyond In-Distribution Success: Scaling Curves of CoT Granularity for Language Model Generalizationcited by 5
  23. 2021Autoencoder-based Semantic Novelty Detection: Towards Dependable AI-based Systemscited by 4
  24. 2024Exponential Expressivity of ReLUk Neural Networks on Gevrey Classes with Point Singularitiescited by 3
  25. 2025Large Language and Reasoning Models are Shallow Disjunctive Reasonerscited by 3
  26. 2018Deep Unsupervised Learning of Visual Similaritiescited by 2
  27. 2022TransDrift: Modeling Word-Embedding Drift using Transformercited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lenscited by 43
  2. 2026Seeing Is Believing? A Benchmark for Multimodal Large Language Models on Visual Illusions and Anomaliescited by 8
  3. 2025Can GRPO Help LLMs Transcend Their Pretraining Origin?cited by 5
  4. 2026On the Out-of-Distribution Generalization of Reasoning in Multimodal LLMs for Simple Visual Planning Taskscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026At the Edge of Understanding: Sparse Autoencoders Trace The Limits of Transformer Generalization
  2. 2025Unlocking Out-of-Distribution Generalization in Transformers via Recursive Latent Space Reasoning
  3. 2022Achieving and Understanding Out-of-Distribution Generalization in Systematic Reasoning in Small-Scale Transformers
  4. 2024Out-of-distribution generalization via composition: a lens through induction heads in Transformers
  5. 2025Measuring Uncertainty in Transformer Circuits with Effective Information Consistency
  6. 2026A Sharper Picture of Generalization in Transformers
  7. 2024Principled Understanding of Generalization for Generative Transformer Models in Arithmetic Reasoning Tasks
  8. 2024Do Efficient Transformers Really Save Computation?