Transformers as Multi-task Learners: Decoupling Features in Hidden Markov Models
Not on the map. Why?
Abstract (arXiv)
Transformer based models have shown remarkable capabilities in sequence learning across a wide range of tasks, often performing well on specific task by leveraging input-output examples. Despite their empirical success, a comprehensive theoretical understanding of this phenomenon remains limited. In this work, we investigate the layerwise behavior of Transformers to uncover the mechanisms underlying their multi-task generalization ability. Taking explorations on a typical sequence model, i.e, Hidden Markov Models, which are fundamental to many language tasks, we observe that: first, lower layers of Transformers focus on extracting feature representations, primarily influenced by neighboring tokens; second, on the upper layers, features become decoupled, exhibiting a high degree of time disentanglement. Building on these empirical insights, we provide theoretical analysis for the expressiveness power of Transformers. Our explicit constructions align closely with empirical observations, providing theoretical support for the Transformer's effectiveness and efficiency on sequence learning across diverse tasks.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 34 of its 49 references that are in the map. First posted on arXiv 2 Jun 2025.
Builds on
34 of 49 references.
References · 34
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023GPT-4 Technical Reportcited by 18,396
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2022Emergent Abilities of Large Language Modelscited by 2,512
- 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
- 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 938
- 2022Transformers learn in-context by gradient descentcited by 732
- 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
- 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
- 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 298
22 more
- 2022Transformers Learn Shortcuts to Automatacited by 268
- 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 233
- 2022Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizerscited by 204
- 2023One Step of Gradient Descent is Provably the Optimal In-Context Learner with One Layer of Linear Self-Attentioncited by 170
- 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 129
- 2024How Transformers Learn Causal Structure with Gradient Descentcited by 118
- 2024Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Processcited by 108
- 2022When Is Partially Observable Reinforcement Learning Not Scary?cited by 100
- 2023Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcited by 83
- 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 75
- 2022GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyondcited by 46
- 2022PAC Reinforcement Learning for Predictive State Representationscited by 43
- 2023A Latent Space Theory for Emergent Abilities in Large Language Modelscited by 43
- 2022Provably Efficient Reinforcement Learning in Partially Observable Dynamical Systemscited by 35
- 2022Corruption-Robust Algorithms with Uncertainty Weighting for Nonlinear Contextual Bandits and Markov Decision Processescited by 33
- 2024How do Transformers perform In-Context Autoregressive Learning?cited by 17
- 2022Embed to Control Partially Observed Systems: Representation Learning with Provable Sample Efficiencycited by 16
- 2019Learning higher-order sequential structure with cloned HMMscited by 10
- 2022Low-Rank Constraints for Fast Inference in Structured Modelscited by 8
- 2023Provably Efficient Representation Learning with Tractable Planning in Low-Rank POMDPcited by 8
- 2023Provably Efficient High-Dimensional Bandit Learning with Batched Feedbackscited by 3
- 2025Transformers and Their Roles as Time Series Foundation Modelscited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.