# UMIE: Unified Multimodal Information Extraction with Instruction Tuning

arXiv: https://arxiv.org/abs/2401.03082 · HTML: https://mlsanity.com/paper/2401.03082/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 43 references that are in the map.
- First posted on arXiv 5 Jan 2024.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Unified Structure Generation for Universal Information Extraction](https://mlsanity.com/paper/2203.12277/) (2022)
- [CLIP-Event: Connecting Text and Images with Event Structures](https://mlsanity.com/paper/2201.05078/) (2022)
- [Grounded Situation Recognition](https://mlsanity.com/paper/2003.12058/) (2020)
- [Multi-Modal Knowledge Graph Construction and Application: A Survey](https://mlsanity.com/paper/2202.05786/) (2022)
- [Cross-media Structured Common Space for Multimedia Event Extraction](https://mlsanity.com/paper/2005.02472/) (2020)
- [Universal Information Extraction as Unified Semantic Matching](https://mlsanity.com/paper/2301.03282/) (2023)
- [Aligning Instruction Tasks Unlocks Large Language Models as Zero-Shot Relation Extractors](https://mlsanity.com/paper/2305.11159/) (2023)
- [Multimodal Named Entity Recognition for Short Social Media Posts](https://mlsanity.com/paper/1802.07862/) (2018)
- [RpBERT: A Text-image Relation Propagation-based BERT Model for Multimodal NER](https://mlsanity.com/paper/2102.02967/) (2021)
- [Named Entity and Relation Extraction with Multi-Modal Retrieval](https://mlsanity.com/paper/2212.01612/) (2022)
- [UnifieR: A Unified Retriever for Large-Scale Retrieval](https://mlsanity.com/paper/2205.11194/) (2022)
- [MNER-QG: An End-to-End MRC framework for Multimodal Named Entity Recognition with Query Grounding](https://mlsanity.com/paper/2211.14739/) (2022)
- [LED: Lexicon-Enlightened Dense Retriever for Large-Scale Retrieval](https://mlsanity.com/paper/2208.13661/) (2022)
- [Flat Multi-modal Interaction Transformer for Named Entity Recognition](https://mlsanity.com/paper/2208.11039/) (2022)
- [Learning Implicit Entity-object Relations by Bidirectional Generative Alignment for Multimodal NER](https://mlsanity.com/paper/2308.02570/) (2023)
- [Multimodal Relation Extraction with Cross-Modal Retrieval and Synthesis](https://mlsanity.com/paper/2305.16166/) (2023)
- [Integrating Large Pre-trained Models into Multimodal Named Entity Recognition with Evidential Fusion](https://mlsanity.com/paper/2306.16991/) (2023)

## Built on by (most cited first, 17 of 17)

- [Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey](https://mlsanity.com/paper/2402.05391/) (2024)
- [MatViX: Multimodal Information Extraction from Visually Rich Articles](https://mlsanity.com/paper/2410.20494/) (2024)
- [Collaborative Multi-LoRA Experts with Achievement-based Multi-Tasks Loss for Unified Multimodal Information Extraction](https://mlsanity.com/paper/2505.06303/) (2025)
- [MMUTF: Multimodal Multimedia Event Argument Extraction with Unified Template Filling](https://mlsanity.com/paper/2406.12420/) (2024)
- [FlexCache: Flexible Approximate Cache System for Video Diffusion](https://mlsanity.com/paper/2501.04012/) (2024)
- [M3Retrieve: Benchmarking Multimodal Retrieval for Medicine](https://mlsanity.com/paper/2510.06888/) (2025)
- [Event Extraction in Large Language Model](https://mlsanity.com/paper/2512.19537/) (2025)
- [DongbaMIE: A Multimodal Information Extraction Dataset for Evaluating Semantic Understanding of Dongba Pictograms](https://mlsanity.com/paper/2503.03644/) (2025)
- [Benchmarking and Improving LVLMs on Event Extraction from Multimedia Documents](https://mlsanity.com/paper/2509.12876/) (2025)
- [Label Drop for Multi-Aspect Relation Modeling in Universal Information Extraction](https://mlsanity.com/paper/2502.12614/) (2025)
- [M-MRE: Extending the Mutual Reinforcement Effect to Multimodal Information Extraction](https://mlsanity.com/paper/2504.17353/) (2025)
- [Integrating gender inclusivity into large language models via instruction tuning](https://mlsanity.com/paper/2508.18466/) (2025)
- [RMPL: Relation-aware Multi-task Progressive Learning with Stage-wise Training for Multimedia Event Extraction](https://mlsanity.com/paper/2602.13748/) (2026)
- [ECHO: Event-Centric Hypergraph Operations via Multi-Agent Collaboration for Multimedia Event Extraction](https://mlsanity.com/paper/2603.06683/) (2026)
- [A Multimodal Text- and Graph-Based Approach for Open-Domain Event Extraction from Documents](https://mlsanity.com/paper/2604.21885/) (2026)
- [SAMA: Semantic Anchor-aligned Augmentation for Unified Low-Resource Multimodal Information Extraction](https://mlsanity.com/paper/2606.18780/) (2026)
- [Evaluation Pitfalls and Challenges in Multimedia Event Extraction](https://mlsanity.com/paper/2606.26775/) (2026)

## Abstract (arXiv)

Multimodal information extraction (MIE) gains significant attention as the popularity of multimedia content increases. However, current MIE methods often resort to using task-specific model structures, which results in limited generalizability across tasks and underutilizes shared knowledge across MIE tasks. To address these issues, we propose UMIE, a unified multimodal information extractor to unify three MIE tasks as a generation problem using instruction tuning, being able to effectively extract both textual and visual mentions. Extensive experiments show that our single UMIE outperforms various state-of-the-art (SoTA) methods across six MIE datasets on three tasks. Furthermore, in-depth analysis demonstrates UMIE's strong generalization in the zero-shot setting, robustness to instruction variants, and interpretability. Our research serves as an initial step towards a unified MIE model and initiates the exploration into both instruction tuning and large language models within the MIE domain. Our code, data, and model are available at https://github.com/ZUCC-AI/UMIE