# Do Transformers Have the Ability for Periodicity Generalization?

arXiv: https://arxiv.org/abs/2601.22690 · HTML: https://mlsanity.com/paper/2601.22690/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 43 references that are in the map.
- First posted on arXiv 30 Jan 2026.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022)
- [Towards Out-Of-Distribution Generalization: A Survey](https://mlsanity.com/paper/2108.13624/) (2021)
- [ETSformer: Exponential Smoothing Transformers for Time-series Forecasting](https://mlsanity.com/paper/2202.01381/) (2022)
- [A Formal Framework for Understanding Length Generalization in Transformers](https://mlsanity.com/paper/2410.02140/) (2024)
- [How Far Are We From AGI: Are LLMs All We Need?](https://mlsanity.com/paper/2405.10313/) (2024)
- [Case-Based or Rule-Based: How Do Transformers Do the Math?](https://mlsanity.com/paper/2402.17709/) (2024)
- [Language Models Need Inductive Biases to Count Inductively](https://mlsanity.com/paper/2405.20131/) (2024)
- [A Survey on Evaluation of Out-of-Distribution Generalization](https://mlsanity.com/paper/2403.01874/) (2024)
- [Out-of-Distribution Generalization in Time Series: A Survey](https://mlsanity.com/paper/2503.13868/) (2025)
- [A Survey on Compositional Learning of AI Models: Theoretical and Experimental Practices](https://mlsanity.com/paper/2406.08787/) (2024)
- [Counting and Algorithmic Generalization with Transformers](https://mlsanity.com/paper/2310.08661/) (2023)
- [Beyond Single-Task: Robust Multi-Task Length Generalization for LLMs](https://mlsanity.com/paper/2502.11525/) (2025)
- [Unlocking Out-of-Distribution Generalization in Transformers via Recursive Latent Space Reasoning](https://mlsanity.com/paper/2510.14095/) (2025)
- [AI-Newton: A Concept-Driven Physical Law Discovery System without Prior Physical Knowledge](https://mlsanity.com/paper/2504.01538/) (2025)
- [Revisiting Generalization Across Difficulty Levels: It's Not So Easy](https://mlsanity.com/paper/2511.21692/) (2025)
- [AI Mathematician: Towards Fully Automated Frontier Mathematical Research](https://mlsanity.com/paper/2505.22451/) (2025)
- [TFKAN: Time-Frequency KAN for Long-Term Time Series Forecasting](https://mlsanity.com/paper/2506.12696/) (2025)
- [Achieving and Understanding Out-of-Distribution Generalization in Systematic Reasoning in Small-Scale Transformers](https://mlsanity.com/paper/2210.03275/) (2022)
- [Reasoning is Periodicity? Improving Large Language Models Through Effective Periodicity Modeling](https://mlsanity.com/paper/2502.21309/) (2025)
- [Learning by Analogy: A Causal Framework for Composition Generalization](https://mlsanity.com/paper/2512.10669/) (2025)

## Abstract (arXiv)

Large language models (LLMs) based on the Transformer have demonstrated strong performance across diverse tasks. However, current models still exhibit substantial limitations in out-of-distribution (OOD) generalization compared with humans. We investigate this gap through periodicity, one of the basic OOD scenarios. Periodicity captures invariance amid variation. Periodicity generalization represents a model's ability to extract periodic patterns from training data and generalize to OOD scenarios. We introduce a unified interpretation of periodicity from the perspective of abstract algebra and reasoning, including both single and composite periodicity, to explain why Transformers struggle to generalize periodicity. Then we construct Coper about composite periodicity, a controllable generative benchmark with two OOD settings, Hollow and Extrapolation. Experiments reveal that periodicity generalization in Transformers is limited, where models can memorize periodic data during training, but cannot generalize to unseen composite periodicity. We release the source code to support future research.