# Attention Bootstrapping for Multi-Modal Test-Time Adaptation

arXiv: https://arxiv.org/abs/2503.02221 · HTML: https://mlsanity.com/paper/2503.02221/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 56 references that are in the map.
- First posted on arXiv 4 Mar 2025.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [Efficient Test-Time Model Adaptation without Forgetting](https://mlsanity.com/paper/2204.02610/) (2022)
- [Towards Stable Test-Time Adaptation in Dynamic Wild World](https://mlsanity.com/paper/2302.12400/) (2023)
- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [Multi-Modal Fusion Transformer for End-to-End Autonomous Driving](https://mlsanity.com/paper/2104.09224/) (2021)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts](https://mlsanity.com/paper/2303.15361/) (2023)
- [Balanced Multimodal Learning via On-the-fly Gradient Modulation](https://mlsanity.com/paper/2203.15332/) (2022)
- [SMIL: Multimodal Learning with Severely Missing Modality](https://mlsanity.com/paper/2103.05677/) (2021)
- [Universal Source-Free Domain Adaptation](https://mlsanity.com/paper/2004.04393/) (2020)
- [Are Multimodal Transformers Robust to Missing Modality?](https://mlsanity.com/paper/2204.05454/) (2022)
- [Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2102.04830/) (2021)
- [Source-Free Domain Adaptation for Semantic Segmentation](https://mlsanity.com/paper/2103.16372/) (2021)
- [PMR: Prototypical Modal Rebalance for Multimodal Learning](https://mlsanity.com/paper/2211.07089/) (2022)
- [Entropy is not Enough for Test-Time Adaptation: From the Perspective of Disentangled Factors](https://mlsanity.com/paper/2403.07366/) (2024)
- [Federated Learning on Non-IID Graphs via Structural Knowledge Sharing](https://mlsanity.com/paper/2211.13009/) (2022)
- [TransRefer3D: Entity-and-Relation Aware Transformer for Fine-Grained 3D Visual Grounding](https://mlsanity.com/paper/2108.02388/) (2021)
- [Improving Multispectral Pedestrian Detection by Addressing Modality Imbalance Problems](https://mlsanity.com/paper/2008.03043/) (2020)
- [MM-TTA: Multi-Modal Test-Time Adaptation for 3D Semantic Segmentation](https://mlsanity.com/paper/2204.12667/) (2022)
- [Back to the Source: Diffusion-Driven Test-Time Adaptation](https://mlsanity.com/paper/2207.03442/) (2022)
- [Test-Time Adaptation to Distribution Shift by Confidence Maximization and Input Transformation](https://mlsanity.com/paper/2106.14999/) (2021)
- [Guiding Pseudo-labels with Uncertainty Estimation for Source-free Unsupervised Domain Adaptation](https://mlsanity.com/paper/2303.03770/) (2023)
- [Target-Driven Structured Transformer Planner for Vision-Language Navigation](https://mlsanity.com/paper/2207.11201/) (2022)
- [Source-Free Domain Adaptation with Frozen Multimodal Foundation Model](https://mlsanity.com/paper/2311.16510/) (2023)
- [Towards Good Practices for Missing Modality Robust Action Recognition](https://mlsanity.com/paper/2211.13916/) (2022)
- [MixNorm: Test-Time Adaptation Through Online Normalization Estimation](https://mlsanity.com/paper/2110.11478/) (2021)
- [Improved Test-Time Adaptation for Domain Generalization](https://mlsanity.com/paper/2304.04494/) (2023)
- [Towards Open-Set Test-Time Adaptation Utilizing the Wisdom of Crowds in Entropy Minimization](https://mlsanity.com/paper/2308.06879/) (2023)

## Built on by (most cited first, 5 of 5)

- [Multifaceted Evaluation of Audio-Visual Capability for MLLMs: Effectiveness, Efficiency, Generalizability and Robustness](https://mlsanity.com/paper/2504.16936/) (2025)
- [Decoupling Stability and Plasticity for Multi-Modal Test-Time Adaptation](https://mlsanity.com/paper/2603.00574/) (2026)
- [PromptDyG: Test-Time Prompt Adaptation on Dynamic Graphs](https://mlsanity.com/paper/2606.22914/) (2026)
- [Test-Time Adaptation for Tactile-Vision-Language Models](https://mlsanity.com/paper/2602.15873/) (2026)
- [Multi-modal Test-time Adaptation via Adaptive Probabilistic Gaussian Calibration](https://mlsanity.com/paper/2604.19093/) (2026)

## Abstract (arXiv)

Test-time adaptation aims to adapt a well-trained model to potential distribution shifts at test time using only unlabeled test data, without access to the original training data. While previous efforts mainly focus on a single modality, test-time distribution shift in the multi-modal setting is more complex and calls for new solutions. This paper tackles the problem of multi-modal test-time adaptation by proposing a novel method named Attention Bootstrapping with Principal Entropy Minimization (ABPEM). We observe that test-time distribution shift causes misalignment across modalities, leading to a large gap between intra-modality discrepancies (measured by self-attention) and inter-modality discrepancies (measured by cross-attention). We name this the attention gap. This attention gap widens with more severe distribution shifts, hindering effective modality fusion. To mitigate this attention gap and encourage better modality fusion, we propose attention bootstrapping that promotes cross-attention with the guidance of self-attention. Moreover, to reduce the gradient noise in the commonly-used entropy minimization, we adopt principal entropy minimization, a refinement of entropy minimization that reduces gradient noise by focusing on the principal parts of entropy, excluding less reliable gradient information. Extensive experiments on the benchmarks validate the effectiveness of the proposed ABPEM in comparison with competing baselines.