# Cross-media Structured Common Space for Multimedia Event Extraction

arXiv: https://arxiv.org/abs/2005.02472 · HTML: https://mlsanity.com/paper/2005.02472/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 May 2020.

## Built on by (most cited first, 30 of 61)

- [Translation between Molecules and Natural Language](https://mlsanity.com/paper/2204.11817/) (2022)
- [CLIP-Event: Connecting Text and Images with Event Structures](https://mlsanity.com/paper/2201.05078/) (2022)
- [Multi-Modal Knowledge Graph Construction and Application: A Survey](https://mlsanity.com/paper/2202.05786/) (2022)
- [Visual Semantic Role Labeling for Video Understanding](https://mlsanity.com/paper/2104.00990/) (2021)
- [Multimodal Automated Fact-Checking: A Survey](https://mlsanity.com/paper/2305.13507/) (2023)
- [Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning](https://mlsanity.com/paper/2312.10160/) (2023)
- [Language Models Can Improve Event Prediction by Few-Shot Abductive Reasoning](https://mlsanity.com/paper/2305.16646/) (2023)
- [TextEE: Benchmark, Reevaluation, Reflections, and Future Challenges in Event Extraction](https://mlsanity.com/paper/2311.09562/) (2023)
- [Joint Multimedia Event Extraction from Video and Article](https://mlsanity.com/paper/2109.12776/) (2021)
- [MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding](https://mlsanity.com/paper/2112.10728/) (2021)
- [What is Event Knowledge Graph: A Survey](https://mlsanity.com/paper/2112.15280/) (2021)
- [EventPlus: A Temporal Event Understanding Pipeline](https://mlsanity.com/paper/2101.04922/) (2021)
- [UMIE: Unified Multimodal Information Extraction with Instruction Tuning](https://mlsanity.com/paper/2401.03082/) (2024)
- [Video Question Answering with Phrases via Semantic Roles](https://mlsanity.com/paper/2104.03762/) (2021)
- [ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representation](https://mlsanity.com/paper/2311.13258/) (2023)
- [A Survey on Deep Learning Event Extraction: Approaches and Applications](https://mlsanity.com/paper/2107.02126/) (2021)
- [A Survey on Integrated Sensing, Communication, and Computation](https://mlsanity.com/paper/2408.08074/) (2024)
- [Training Multimedia Event Extraction With Generated Images and Captions](https://mlsanity.com/paper/2306.08966/) (2023)
- [Information Extraction in Low-Resource Scenarios: Survey and Perspective](https://mlsanity.com/paper/2202.08063/) (2022)
- [Video Event Extraction via Tracking Visual States of Arguments](https://mlsanity.com/paper/2211.01781/) (2022)
- [MultiVENT: Multilingual Videos of Events with Aligned Natural Text](https://mlsanity.com/paper/2307.03153/) (2023)
- [Improving Event Definition Following For Zero-Shot Event Detection](https://mlsanity.com/paper/2403.02586/) (2024)
- [Event Detection from Social Media for Epidemic Prediction](https://mlsanity.com/paper/2404.01679/) (2024)
- [SPEED++: A Multilingual Event Extraction Framework for Epidemic Prediction and Preparedness](https://mlsanity.com/paper/2410.18393/) (2024)
- [Collaborative Multi-LoRA Experts with Achievement-based Multi-Tasks Loss for Unified Multimodal Information Extraction](https://mlsanity.com/paper/2505.06303/) (2025)
- [Understanding Social Media Cross-Modality Discourse in Linguistic Space](https://mlsanity.com/paper/2302.13311/) (2023)
- [Multimodal Question Answering for Unified Information Extraction](https://mlsanity.com/paper/2310.03017/) (2023)
- [MMUTF: Multimodal Multimedia Event Argument Extraction with Unified Template Filling](https://mlsanity.com/paper/2406.12420/) (2024)
- [Event Extraction in Large Language Model](https://mlsanity.com/paper/2512.19537/) (2025)
- [Fine-Grained Visual Entailment](https://mlsanity.com/paper/2203.15704/) (2022)

## Abstract (arXiv)

We introduce a new task, MultiMedia Event Extraction (M2E2), which aims to extract events and their arguments from multimedia documents. We develop the first benchmark and collect a dataset of 245 multimedia news articles with extensively annotated events and arguments. We propose a novel method, Weakly Aligned Structured Embedding (WASE), that encodes structured representations of semantic information from textual and visual data into a common embedding space. The structures are aligned across modalities by employing a weakly supervised training strategy, which enables exploiting available resources without explicit cross-media annotation. Compared to uni-modal state-of-the-art methods, our approach achieves 4.0% and 9.8% absolute F-score gains on text event argument role labeling and visual event extraction. Compared to state-of-the-art multimedia unstructured representations, we achieve 8.3% and 5.0% absolute F-score gains on multimedia event extraction and argument role labeling, respectively. By utilizing images, we extract 21.4% more event mentions than traditional text-only methods.