Multi-layer Cross-attention is Provably Optimal for Multi-modal In-context Learning
Not on the map. Why?
Abstract (arXiv)
Recent progress has rapidly advanced our understanding of the mechanisms underlying in-context learning in modern attention-based neural networks. However, existing results focus exclusively on unimodal data; in contrast, the theoretical underpinnings of in-context learning for multi-modal data remain poorly understood. We introduce a mathematically tractable framework for studying multi-modal learning and explore when transformer-like architectures can recover Bayes-optimal performance in-context. To model multi-modal problems, we assume the observed data arises from a latent factor model. Our first result comprises a negative take on expressibility: we prove that single-layer, linear self-attention fails to recover the Bayes-optimal predictor uniformly over the task distribution. To address this limitation, we introduce a novel, linearized cross-attention mechanism, which we study in the regime where both the number of cross-attention layers and the context length are large. We show that this cross-attention mechanism is provably Bayes optimal when optimized using gradient flow. Our results underscore the benefits of depth for in-context learning and establish the provable utility of cross-attention for multi-modal distributions.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 48 of its 53 references that are in the map. First posted on arXiv 4 Feb 2026.
Builds on
48 of 53 references.
References · 48
- 2017Attention Is All You Needcited by 59,932
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
- 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
- 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
- 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
- 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
- 2020On Layer Normalization in the Transformer Architecturecited by 906
- 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
- 2022Transformers learn in-context by gradient descentcited by 714
- 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
36 more
- 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
- 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
- 2019Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processingcited by 411
- 2023Trained Transformers Learn Linear Models In-Contextcited by 331
- 2023Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selectioncited by 291
- 2023Transformers learn to implement preconditioned gradient descent for in-context learningcited by 274
- 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
- 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 229
- 2023Transformers as Algorithms: Generalization and Stability in In-context Learningcited by 220
- 2023Handbook of Convergence Theorems for (Stochastic) Gradient Methodscited by 158
- 2018Contextual Stochastic Block Modelscited by 134
- 2023In-Context Convergence of Transformerscited by 112
- 2023How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?cited by 93
- 2018CentralNet: a Multilayer Approach for Multimodal Fusioncited by 86
- 2024Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimalitycited by 71
- 2024Asymptotic theory of in-context learning by linear attentioncited by 54
- 2023On the Role of Attention in Prompt-tuningcited by 49
- 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 44
- 2024A phase transition between positional and semantic learning in a solvable model of dot-product attentioncited by 30
- 2024In-Context Learning of a Linear Transformer Block: Benefits of the MLP Component and One-Step GD Initializationcited by 30
- 2021Cooperative learning for multiview analysiscited by 19
- 2025Continual Multimodal Contrastive Learningcited by 19
- 2024On the Training Convergence of Transformers for In-Context Classification of Gaussian Mixturescited by 15
- 2025Fundamental limits of learning in sequence multi-index models and deep attention networks: High-dimensional asymptotics and sharp thresholdscited by 12
- 2025Asymptotics of SGD in Sequence-Single Index Models and Single-Layer Attention Networkscited by 9
- 2024Fundamental limits of community detection from multi-view data: multi-layer, dynamic and partially labeled block modelscited by 7
- 2025Multi-modal contrastive learning adapts to intrinsic dimensions of shared latent variablescited by 7
- 2026Efficient and Minimax Optimal In-context Nonparametric Regression with Transformerscited by 7
- 2025High-Dimensional Analysis of Single-Layer Attention for Sparse-Token Classificationcited by 6
- 2025Pretrain-Test Task Alignment Governs Generalization in In-Context Learningcited by 6
- 2024Multimodal data integration and cross-modal querying via orchestrated approximate message passingcited by 5
- 2024A spectral method for multi-view subspace learning using the product of projectionscited by 5
- 2024Optimal thresholds and algorithms for a model of multi-modal learning in high dimensionscited by 4
- 2025Spectral Thresholds in Correlated Spiked Models and Fundamental Limits of Partial Least Squarescited by 4
- 2025Attention-based clusteringcited by 3
- 2024Contrastive Learning on Multimodal Analysis of Electronic Health Recordscited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.