# REMOTE: A Unified Multimodal Relation Extraction Framework with Multilevel Optimal Transport and Mixture-of-Experts

arXiv: https://arxiv.org/abs/2509.04844 · HTML: https://mlsanity.com/paper/2509.04844/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts.
- Builds on 11 of its 18 references that are in the map.
- First posted on arXiv 5 Sep 2025.

## Placed in

- Mixture-of-experts / [Multimodal MoE models](https://mlsanity.com/moe/multimodal-moe-models/)

## Builds on (most cited first, 11 of 11)

- [TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task](https://mlsanity.com/paper/2004.14855/) (2020)
- [Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion](https://mlsanity.com/paper/2205.02357/) (2022)
- [Information Screening whilst Exploiting! Multimodal Relation Extraction with Feature Denoising and Multimodal Topic Modeling](https://mlsanity.com/paper/2305.11719/) (2023)
- [Prompting ChatGPT in MNER: Enhanced Multimodal Named Entity Recognition with Auxiliary Refined Knowledge](https://mlsanity.com/paper/2305.12212/) (2023)
- [Multimodal Relation Extraction with Cross-Modal Retrieval and Synthesis](https://mlsanity.com/paper/2305.16166/) (2023)
- [Dual-Gated Fusion with Prefix-Tuning for Multi-Modal Relation Extraction](https://mlsanity.com/paper/2306.11020/) (2023)
- [LLMs as Bridges: Reformulating Grounded Multimodal Named Entity Recognition](https://mlsanity.com/paper/2402.09989/) (2024)
- [On Analyzing the Role of Image for Visual-enhanced Relation Extraction](https://mlsanity.com/paper/2211.07504/) (2022)
- [Prompt Me Up: Unleashing the Power of Alignments for Multimodal Entity and Relation Extraction](https://mlsanity.com/paper/2310.16822/) (2023)
- [MORE: A Multimodal Object-Entity Relation Extraction Dataset with a Benchmark Evaluation](https://mlsanity.com/paper/2312.09753/) (2023)
- [Shapley Value-based Contrastive Alignment for Multimodal Information Extraction](https://mlsanity.com/paper/2407.17854/) (2024)

## Built on by (most cited first, 2 of 2)

- [Feature-level Interaction Explanations in Multimodal Transformers](https://mlsanity.com/paper/2603.13326/) (2026)
- [UG-UMRE: Uncertainty-Guided Modality Augmentation and Distributional Calibration for Unified Multimodal Relation Extraction](https://mlsanity.com/paper/2608.04949/) (2026)

## Abstract (arXiv)

Multimodal relation extraction (MRE) is a crucial task in the fields of Knowledge Graph and Multimedia, playing a pivotal role in multimodal knowledge graph construction. However, existing methods are typically limited to extracting a single type of relational triplet, which restricts their ability to extract triplets beyond the specified types. Directly combining these methods fails to capture dynamic cross-modal interactions and introduces significant computational redundancy. Therefore, we propose a novel textitunified multimodal Relation Extraction framework with Multilevel Optimal Transport and mixture-of-Experts, termed REMOTE, which can simultaneously extract intra-modal and inter-modal relations between textual entities and visual objects. To dynamically select optimal interaction features for different types of relational triplets, we introduce mixture-of-experts mechanism, ensuring the most relevant modality information is utilized. Additionally, considering that the inherent property of multilayer sequential encoding in existing encoders often leads to the loss of low-level information, we adopt a multilevel optimal transport fusion module to preserve low-level features while maintaining multilayer encoding, yielding more expressive representations. Correspondingly, we also create a Unified Multimodal Relation Extraction (UMRE) dataset to evaluate the effectiveness of our framework, encompassing diverse cases where the head and tail entities can originate from either text or image. Extensive experiments show that REMOTE effectively extracts various types of relational triplets and achieves state-of-the-art performanc on almost all metrics across two other public MRE datasets. We release our resources at https://github.com/Nikol-coder/REMOTE.