# Few-Shot Relation Extraction with Hybrid Visual Evidence

arXiv: https://arxiv.org/abs/2403.00724 · HTML: https://mlsanity.com/paper/2403.00724/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 37 of its 76 references that are in the map.
- First posted on arXiv 1 Mar 2024.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [YOLOv4: Optimal Speed and Accuracy of Object Detection](https://mlsanity.com/paper/2004.10934/) (2020)
- [Multimodal Few-Shot Learning with Frozen Language Models](https://mlsanity.com/paper/2106.13884/) (2021)
- [Deep Modular Co-Attention Networks for Visual Question Answering](https://mlsanity.com/paper/1906.10770/) (2019)
- [Few-Shot Learning with Graph Neural Networks](https://mlsanity.com/paper/1711.04043/) (2017)
- [Bilinear Attention Networks](https://mlsanity.com/paper/1805.07932/) (2018)
- [Matching the Blanks: Distributional Similarity for Relation Learning](https://mlsanity.com/paper/1906.03158/) (2019)
- [Meta Networks](https://mlsanity.com/paper/1703.00837/) (2017)
- [FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation](https://mlsanity.com/paper/1810.10147/) (2018)
- [MMTM: Multimodal Transfer Module for CNN Fusion](https://mlsanity.com/paper/1911.08670/) (2019)
- [Multimodal Contrastive Training for Visual Representation Learning](https://mlsanity.com/paper/2104.12836/) (2021)
- [Deep Multimodal Fusion by Channel Exchanging](https://mlsanity.com/paper/2011.05005/) (2020)
- [Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models](https://mlsanity.com/paper/2301.06267/) (2023)
- [CentralNet: a Multilayer Approach for Multimodal Fusion](https://mlsanity.com/paper/1808.07275/) (2018)
- [BLOCK: Bilinear Superdiagonal Fusion for Visual Question Answering and Visual Relationship Detection](https://mlsanity.com/paper/1902.00038/) (2019)
- [Few-shot Relation Extraction via Bayesian Meta-learning on Relation Graphs](https://mlsanity.com/paper/2007.02387/) (2020)
- [Multi-Level Matching and Aggregation Network for Few-Shot Relation Classification](https://mlsanity.com/paper/1906.06678/) (2019)
- [Exploring Task Difficulty for Few-Shot Relation Extraction](https://mlsanity.com/paper/2109.05473/) (2021)
- [Neural Snowball for Few-Shot Relation Learning](https://mlsanity.com/paper/1908.11007/) (2019)
- [Continual Few-shot Relation Learning via Embedding Space Regularization and Data Augmentation](https://mlsanity.com/paper/2203.02135/) (2022)
- [MEWL: Few-shot multimodal word learning with referential uncertainty](https://mlsanity.com/paper/2306.00503/) (2023)
- [MapRE: An Effective Semantic Mapping Approach for Low-resource Relation Extraction](https://mlsanity.com/paper/2109.04108/) (2021)
- [Entity Concept-enhanced Few-shot Relation Extraction](https://mlsanity.com/paper/2106.02401/) (2021)
- [Joint Multimodal Entity-Relation Extraction Based on Edge-enhanced Graph Alignment Network and Word-pair Relation Tagging](https://mlsanity.com/paper/2211.15028/) (2022)
- [A Simple yet Effective Relation Information Guided Approach for Few-Shot Relation Extraction](https://mlsanity.com/paper/2205.09536/) (2022)
- [Few-shot Multimodal Sentiment Analysis based on Multimodal Probabilistic Fusion Prompts](https://mlsanity.com/paper/2211.06607/) (2022)

## Abstract (arXiv)

The goal of few-shot relation extraction is to predict relations between name entities in a sentence when only a few labeled instances are available for training. Existing few-shot relation extraction methods focus on uni-modal information such as text only. This reduces performance when there are no clear contexts between the name entities described in text. We propose a multi-modal few-shot relation extraction model (MFS-HVE) that leverages both textual and visual semantic information to learn a multi-modal representation jointly. The MFS-HVE includes semantic feature extractors and multi-modal fusion components. The MFS-HVE semantic feature extractors are developed to extract both textual and visual features. The visual features include global image features and local object features within the image. The MFS-HVE multi-modal fusion unit integrates information from various modalities using image-guided attention, object-guided attention, and hybrid feature attention to fully capture the semantic interaction between visual regions of images and relevant texts. Extensive experiments conducted on two public datasets demonstrate that semantic visual information significantly improves the performance of few-shot relation prediction.