# Eureka-Moments in Transformers: Multi-Step Tasks Reveal Softmax Induced Optimization Problems arXiv: https://arxiv.org/abs/2310.12956 ยท HTML: https://mlsanity.com/paper/2310.12956/ ## Key facts - Cited by 14 papers in the map (data through 14 Oct 2026). - Builds on 37 of its 54 references that are in the map. - First posted on arXiv 19 Oct 2023. ## Builds on (most cited first, 30 of 37) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023) - [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020) - [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017) - [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022) - [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022) - [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020) - [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019) - [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019) - [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019) - [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022) - [On Layer Normalization in the Transformer Architecture](https://mlsanity.com/paper/2002.04745/) (2020) - [Progress measures for grokking via mechanistic interpretability](https://mlsanity.com/paper/2301.05217/) (2023) - [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023) - [Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets](https://mlsanity.com/paper/2201.02177/) (2022) - [Are Emergent Abilities of Large Language Models a Mirage?](https://mlsanity.com/paper/2304.15004/) (2023) - [Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth](https://mlsanity.com/paper/2103.03404/) (2021) - [XCiT: Cross-Covariance Image Transformers](https://mlsanity.com/paper/2106.09681/) (2021) - [Learning Deep Transformer Models for Machine Translation](https://mlsanity.com/paper/1906.01787/) (2019) - [Data Distributional Properties Drive Emergent In-Context Learning in Transformers](https://mlsanity.com/paper/2205.05055/) (2022) - [Escaping the Big Data Paradigm with Compact Transformers](https://mlsanity.com/paper/2104.05704/) (2021) - [Information-Theoretic Probing with Minimum Description Length](https://mlsanity.com/paper/2003.12298/) (2020) - [Towards Understanding Grokking: An Effective Theory of Representation Learning](https://mlsanity.com/paper/2205.10343/) (2022) - [Adaptive Input Representations for Neural Language Modeling](https://mlsanity.com/paper/1809.10853/) (2018) - [Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit](https://mlsanity.com/paper/2207.08799/) (2022) - [Stabilizing Transformer Training by Preventing Attention Entropy Collapse](https://mlsanity.com/paper/2303.06296/) (2023) - [Omnigrok: Grokking Beyond Algorithmic Data](https://mlsanity.com/paper/2210.01117/) (2022) - [Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapse](https://mlsanity.com/paper/2206.03126/) (2022) ## Built on by (most cited first, 14 of 14) - [The Power of Noise: Redefining Retrieval for RAG Systems](https://mlsanity.com/paper/2401.14887/) (2024) - [Training Dynamics of In-Context Learning in Linear Attention](https://mlsanity.com/paper/2501.16265/) (2025) - [Improving fine-grained understanding in image-text pre-training](https://mlsanity.com/paper/2401.09865/) (2024) - [A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Language](https://mlsanity.com/paper/2408.12578/) (2024) - [Abrupt Learning in Transformers: A Case Study on Matrix Completion](https://mlsanity.com/paper/2410.22244/) (2024) - [What Happens During the Loss Plateau? Understanding Abrupt Learning in Transformers](https://mlsanity.com/paper/2506.13688/) (2025) - [Quantum Doubly Stochastic Transformers](https://mlsanity.com/paper/2504.16275/) (2025) - [Unpacking Softmax: How Temperature Drives Representation Collapse, Compression, and Generalization](https://mlsanity.com/paper/2506.01562/) (2025) - [Convergent World Representations and Divergent Tasks](https://mlsanity.com/paper/2602.00533/) (2026) - [A Mechanistic Study of Transformers Training Dynamics](https://mlsanity.com/paper/2410.24050/) (2024) - [Cross-modal Full-mode Fine-grained Alignment for Text-to-Image Person Retrieval](https://mlsanity.com/paper/2509.13754/) (2025) - [Algorithm Development in Neural Networks: Insights from the Streaming Parity Task](https://mlsanity.com/paper/2507.09897/) (2025) - [Memory as a cache: Exact context reuse and deletion by construction](https://mlsanity.com/paper/2609.32395/) (2026) - [Pretraining Transformers with Quantized Softmax in Attention](https://mlsanity.com/paper/2609.33591/) (2026) ## Abstract (arXiv) In this work, we study rapid improvements of the training loss in transformers when being confronted with multi-step decision tasks. We found that transformers struggle to learn the intermediate task and both training and validation loss saturate for hundreds of epochs. When transformers finally learn the intermediate task, they do this rapidly and unexpectedly. We call these abrupt improvements Eureka-moments, since the transformer appears to suddenly learn a previously incomprehensible concept. We designed synthetic tasks to study the problem in detail, but the leaps in performance can be observed also for language modeling and in-context learning (ICL). We suspect that these abrupt transitions are caused by the multi-step nature of these tasks. Indeed, we find connections and show that ways to improve on the synthetic multi-step tasks can be used to improve the training of language modeling and ICL. Using the synthetic data we trace the problem back to the Softmax function in the self-attention block of transformers and show ways to alleviate the problem. These fixes reduce the required number of training steps, lead to higher likelihood to learn the intermediate task, to higher final accuracy and training becomes more robust to hyper-parameters.