# Physics-based phenomenological characterization of cross-modal bias in multimodal models

arXiv: https://arxiv.org/abs/2602.20624 · HTML: https://mlsanity.com/paper/2602.20624/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 39 references that are in the map.
- First posted on arXiv 24 Feb 2026.

## Builds on (most cited first, 17 of 17)

- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [Towards Generalist Biomedical AI](https://mlsanity.com/paper/2307.14334/) (2023)
- [MM-LLMs: Recent Advances in MultiModal Large Language Models](https://mlsanity.com/paper/2401.13601/) (2024)
- [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020)
- [A mathematical perspective on Transformers](https://mlsanity.com/paper/2312.10794/) (2023)
- [The emergence of clusters in self-attention dynamics](https://mlsanity.com/paper/2305.05465/) (2023)
- [Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of View](https://mlsanity.com/paper/1906.02762/) (2019)
- [Adversarial Distributional Training for Robust Deep Learning](https://mlsanity.com/paper/2002.05999/) (2020)
- [Emergence of meta-stable clustering in mean-field transformer models](https://mlsanity.com/paper/2410.23228/) (2024)
- [Redesigning the Transformer Architecture with Insights from Multi-particle Dynamical Systems](https://mlsanity.com/paper/2109.15142/) (2021)
- [The Mean-Field Dynamics of Transformers](https://mlsanity.com/paper/2512.01868/) (2025)
- [Calibrating LLM Confidence by Probing Perturbed Representation Stability](https://mlsanity.com/paper/2505.21772/) (2025)
- [Multimodal Foundation Models Exploit Text to Make Medical Image Predictions](https://mlsanity.com/paper/2311.05591/) (2023)
- [Text Takes Over: A Study of Modality Bias in Multimodal Intent Detection](https://mlsanity.com/paper/2508.16122/) (2025)
- [From the New World of Word Embeddings: A Comparative Study of Small-World Lexico-Semantic Networks in LLMs](https://mlsanity.com/paper/2502.11380/) (2025)
- [Why Text Prevails: Vision May Undermine Multimodal Medical Decision Making](https://mlsanity.com/paper/2512.13747/) (2025)

## Built on by (most cited first, 1 of 1)

- [Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes](https://mlsanity.com/paper/2608.05000/) (2026)

## Abstract (arXiv)

The term 'algorithmic fairness' is used to evaluate whether AI models operate fairly in both comparative (where fairness is understood as formal equality, such as "treat like cases as like") and non-comparative (where unfairness arises from the model's inaccuracy, arbitrariness, or inscrutability) contexts. Recent advances in multimodal large language models (MLLMs) are breaking new ground in multimodal understanding, reasoning, and generation; however, we argue that inconspicuous distortions arising from complex multimodal interaction dynamics can lead to systematic bias. The purpose of this position paper is twofold: first, it is intended to acquaint AI researchers with phenomenological explainable approaches that rely on the physical entities that the machine experiences during training/inference, as opposed to the traditional cognitivist symbolic account or metaphysical approaches; second, it is to state that this phenomenological doctrine will be practically useful for tackling algorithmic fairness issues in MLLMs. We develop a surrogate physics-based model that describes transformer dynamics (i.e., semantic network structure and self-/cross-attention) to analyze the dynamics of cross-modal bias in MLLM, which are not fully captured by conventional embedding- or representation-level analyses. We support this position through multi-input diagnostic experiments: 1) perturbation-based analyses of emotion classification using Qwen2.5-Omni and Gemma 3n, and 2) dynamical analysis of Lorenz chaotic time-series prediction through the physical surrogate. Across two architecturally distinct MLLMs, we show that multimodal inputs can reinforce modality dominance rather than mitigate it, as revealed by structured error-attractor patterns under systematic label perturbation, complemented by dynamical analysis.