Alignment Helps Make the Most of Multimodal Data
Not on the map. Why?
Abstract (arXiv)
Political scientists increasingly analyze multimodal data. However, the effective analysis of such data requires aligning information across different modalities. In our paper, we demonstrate the significance of such alignment. Informed by a systematic review of 2,703 papers, we find that political scientists typically do not align their multimodal data. Introducing a decision tree that guides alignment choices, our framework highlights alignment's untapped potential and provides concrete advice in research design and modeling decisions. We illustrate alignment's analytical value through two applications: predicting tonality in U.S. presidential campaign ads and cross-modal querying of German parliamentary speeches to examine responses to the far-right AfD.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 28 of its 105 references that are in the map. First posted on arXiv 14 May 2024.
Builds on
28 of 105 references.
References · 28
- 2020End-to-End Object Detection with Transformerscited by 7,414
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
- 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
- 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
- 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
- 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
- 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
- 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
- 2023WhisperX: Time-Accurate Speech Transcription of Long-Form Audiocited by 326
- 2013Zero-Shot Learning Through Cross-Modal Transfercited by 237
- 2016On Deep Multi-View Representation Learning: Objectives and Optimizationcited by 162
- 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 135
16 more
- 2015Natural Language Object Retrievalcited by 124
- 2020Multi-Modal Domain Adaptation for Fine-Grained Action Recognitioncited by 115
- 2014Deep Fragment Embeddings for Bidirectional Image Sentence Mappingcited by 99
- 20226D Rotation Representation For Unconstrained Head Pose Estimationcited by 64
- 2016Dynamic Memory Networks for Visual and Textual Question Answeringcited by 62
- 2020Does my multimodal model learn cross-modal interactions? It's harder to tell than you might think!cited by 62
- 2018How to Make Causal Inferences Using Textscited by 59
- 2018Visual Coreference Resolution in Visual Dialog using Neural Module Networkscited by 54
- 2021Multimodal Learning using Optimal Transport for Sarcasm and Humor Detectioncited by 37
- 2022MultiViz: Towards Visualizing and Understanding Multimodal Modelscited by 36
- 2021M2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysiscited by 29
- 2016Audio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusioncited by 23
- 2018Matching with Text Data: An Experimental Evaluation of Methods for Matching Documents and of Measuring Match Qualitycited by 21
- 2020Cross-Modal Generalization: Learning in Low Resource Modalities via Meta-Alignmentcited by 16
- 2021Watch Those Words: Video Falsification Detection Using Word-Conditioned Facial Motioncited by 15
- 2021Analyzing Online Political Advertisementscited by 11
Built on by
Citing papers per year: 2025: 1.
1 paper.