# Multimodal Cross-Document Event Coreference Resolution Using Linear Semantic Transfer and Mixed-Modality Ensembles

arXiv: https://arxiv.org/abs/2404.08949 · HTML: https://mlsanity.com/paper/2404.08949/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 59 references that are in the map.
- First posted on arXiv 13 Apr 2024.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Understanding image representations by measuring their equivariance and equivalence](https://mlsanity.com/paper/1411.5908/) (2014)
- [Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring](https://mlsanity.com/paper/1905.01969/) (2019)
- [Linearly Mapping from Image to Text Space](https://mlsanity.com/paper/2209.15162/) (2022)
- [Cross-media Structured Common Space for Multimedia Event Extraction](https://mlsanity.com/paper/2005.02472/) (2020)
- [Multimodal Transformer Networks for End-to-End Video-Grounded Dialogue Systems](https://mlsanity.com/paper/1907.01166/) (2019)
- [Cross-document Coreference Resolution over Predicted Mentions](https://mlsanity.com/paper/2106.01210/) (2021)
- [Resolving Event Coreference with Supervised Representation Learning and Clustering-Oriented Regularization](https://mlsanity.com/paper/1805.10985/) (2018)
- [Joint Multimedia Event Extraction from Video and Article](https://mlsanity.com/paper/2109.12776/) (2021)
- [Generalizing Cross-Document Event Coreference Resolution Across Multiple Corpora](https://mlsanity.com/paper/2011.12249/) (2020)
- [Focus on what matters: Applying Discourse Coherence Theory to Cross Document Coreference](https://mlsanity.com/paper/2110.05362/) (2021)
- [2 * n is better than n2: Decomposing Event Coreference Resolution into Two Tractable Problems](https://mlsanity.com/paper/2305.05672/) (2023)
- [What happens before and after: Multi-Event Commonsense in Event Coreference Resolution](https://mlsanity.com/paper/2302.09715/) (2023)
- [Using Automatically Extracted Minimum Spans to Disentangle Coreference Evaluation from Boundary Detection](https://mlsanity.com/paper/1906.06703/) (2019)
- [Pairwise Representation Learning for Event Coreference](https://mlsanity.com/paper/2010.12808/) (2020)
- [Grounding and Distinguishing Conceptual Vocabulary Through Similarity Learning in Embodied Simulations](https://mlsanity.com/paper/2305.13668/) (2023)

## Built on by (most cited first, 4 of 4)

- [Who Relies More on World Knowledge and Bias for Syntactic Ambiguity Resolution: Humans or LLMs?](https://mlsanity.com/paper/2503.10838/) (2025)
- [Link Prediction for Event Logs in the Process Industry](https://mlsanity.com/paper/2508.09096/) (2025)
- [Piecing Together Cross-Document Coreference Resolution Datasets: Systematic Dataset Analysis and Unification](https://mlsanity.com/paper/2603.00621/) (2026)
- [Evaluation Pitfalls and Challenges in Multimedia Event Extraction](https://mlsanity.com/paper/2606.26775/) (2026)

## Abstract (arXiv)

Event coreference resolution (ECR) is the task of determining whether distinct mentions of events within a multi-document corpus are actually linked to the same underlying occurrence. Images of the events can help facilitate resolution when language is ambiguous. Here, we propose a multimodal cross-document event coreference resolution method that integrates visual and textual cues with a simple linear map between vision and language models. As existing ECR benchmark datasets rarely provide images for all event mentions, we augment the popular ECB+ dataset with event-centric images scraped from the internet and generated using image diffusion models. We establish three methods that incorporate images and text for coreference: 1) a standard fused model with finetuning, 2) a novel linear mapping method without finetuning and 3) an ensembling approach based on splitting mention pairs by semantic and discourse-level difficulty. We evaluate on 2 datasets: the augmented ECB+, and AIDA Phase 1. Our ensemble systems using cross-modal linear mapping establish an upper limit (91.9 CoNLL F1) on ECB+ ECR performance given the preprocessing assumptions used, and establish a novel baseline on AIDA Phase 1. Our results demonstrate the utility of multimodal information in ECR for certain challenging coreference problems, and highlight a need for more multimodal resources in the coreference resolution space.