# One-Versus-Others Attention: Scalable Multimodal Integration for Biomedical Data

arXiv: https://arxiv.org/abs/2307.05435 · HTML: https://mlsanity.com/paper/2307.05435/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 72 references that are in the map.
- First posted on arXiv 11 Jul 2023.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [VisualBERT: A Simple and Performant Baseline for Vision and Language](https://mlsanity.com/paper/1908.03557/) (2019)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [VL-BERT: Pre-training of Generic Visual-Linguistic Representations](https://mlsanity.com/paper/1908.08530/) (2019)
- [VideoBERT: A Joint Model for Video and Language Representation Learning](https://mlsanity.com/paper/1904.01766/) (2019)
- [Multimodal Transformer for Unaligned Multimodal Language Sequences](https://mlsanity.com/paper/1906.00295/) (2019)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text](https://mlsanity.com/paper/2104.11178/) (2021)
- [Attention Bottlenecks for Multimodal Fusion](https://mlsanity.com/paper/2107.00135/) (2021)
- [Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers](https://mlsanity.com/paper/2004.00849/) (2020)
- [ActBERT: Learning Global-Local Video-Text Representations](https://mlsanity.com/paper/2011.07231/) (2020)
- [DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection](https://mlsanity.com/paper/2203.08195/) (2022)
- [Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing](https://mlsanity.com/paper/2301.04558/) (2023)
- [ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data](https://mlsanity.com/paper/2001.07966/) (2020)
- [End-to-end Generative Pretraining for Multimodal Video Captioning](https://mlsanity.com/paper/2201.08264/) (2022)
- [Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training](https://mlsanity.com/paper/2105.11333/) (2021)
- [Multimodal Transformer with Multi-View Visual Representation for Image Captioning](https://mlsanity.com/paper/1905.07841/) (2019)
- [MMBERT: Multimodal BERT Pretraining for Improved Medical VQA](https://mlsanity.com/paper/2104.01394/) (2021)
- [MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images](https://mlsanity.com/paper/2207.07027/) (2022)
- [Benchmarking machine learning models on multi-centre eICU critical care dataset](https://mlsanity.com/paper/1910.00964/) (2019)
- [Deep Orthogonal Fusion: Multimodal Prognostic Biomarker Discovery Integrating Radiology, Pathology, Genomic, and Clinical Data](https://mlsanity.com/paper/2107.00648/) (2021)
- [Multimodal Attention-based Deep Learning for Alzheimer's Disease Diagnosis](https://mlsanity.com/paper/2206.08826/) (2022)
- [Multimodal Sentiment Analysis: Addressing Key Issues and Setting up the Baselines](https://mlsanity.com/paper/1803.07427/) (2018)

## Built on by (most cited first, 6 of 6)

- [PiCME: Pipeline for Contrastive Modality Evaluation and Encoding in the MIMIC Dataset](https://mlsanity.com/paper/2507.03165/) (2025)
- [MoXGATE: Modality-aware cross-attention for multi-omic gastrointestinal cancer sub-type classification](https://mlsanity.com/paper/2506.06980/) (2025)
- [CAAT-EHR: Cross-Attentional Autoregressive Transformer for Multimodal Electronic Health Record Embeddings](https://mlsanity.com/paper/2501.18891/) (2025)
- [A Systematic Review on the Generative AI Applications in Human Medical Genomics](https://mlsanity.com/paper/2508.20275/) (2025)
- [CLARGA: Multimodal Graph Representation Learning over Arbitrary Sets of Modalities](https://mlsanity.com/paper/2512.11901/) (2025)
- [Multimodal QUD: Inquisitive Questions from Scientific Figures](https://mlsanity.com/paper/2604.23733/) (2026)

## Abstract (arXiv)

Multimodal learning models have become increasingly important as they surpass single-modality approaches on diverse tasks ranging from question-answering to autonomous driving. Despite the importance of multimodal learning, existing efforts focus on NLP applications, where the number of modalities is typically less than four (audio, video, text, images). However, data inputs in other domains, such as the medical field, may include X-rays, PET scans, MRIs, genetic screening, clinical notes, and more, creating a need for both efficient and accurate information fusion. Many state-of-the-art models rely on pairwise cross-modal attention, which does not scale well for applications with more than three modalities. For n modalities, computing attention will result in n choose 2 operations, potentially requiring considerable amounts of computational resources. To address this, we propose a new domain-neutral attention mechanism, One-Versus-Others (OvO) attention, that scales linearly with the number of modalities and requires only n attention operations, thus offering a significant reduction in computational complexity compared to existing cross-modal attention algorithms. Using three diverse real-world datasets as well as an additional simulation experiment, we show that our method improves performance compared to popular fusion techniques while decreasing computation costs.