mlsanityData to 7 Oct 2026

Few-Shot Relation Extraction with Hybrid Visual Evidence

arXiv 2403.00724 ↗First posted 1 Mar 2024cs.CL cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

The goal of few-shot relation extraction is to predict relations between name entities in a sentence when only a few labeled instances are available for training. Existing few-shot relation extraction methods focus on uni-modal information such as text only. This reduces performance when there are no clear contexts between the name entities described in text. We propose a multi-modal few-shot relation extraction model (MFS-HVE) that leverages both textual and visual semantic information to learn a multi-modal representation jointly. The MFS-HVE includes semantic feature extractors and multi-modal fusion components. The MFS-HVE semantic feature extractors are developed to extract both textual and visual features. The visual features include global image features and local object features within the image. The MFS-HVE multi-modal fusion unit integrates information from various modalities using image-guided attention, object-guided attention, and hybrid feature attention to fully capture the semantic interaction between visual regions of images and relevant texts. Extensive experiments conducted on two public datasets demonstrate that semantic visual information significantly improves the performance of few-shot relation prediction.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 37 of its 76 references that are in the map. First posted on arXiv 1 Mar 2024.

Builds on

37 of 76 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2017Prototypical Networks for Few-shot Learningcited by 3,001
  6. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,624
  7. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  8. 2019Deep Modular Co-Attention Networks for Visual Question Answeringcited by 338
  9. 2017Few-Shot Learning with Graph Neural Networkscited by 324
  10. 2018Bilinear Attention Networkscited by 277
  11. 2019Matching the Blanks: Distributional Similarity for Relation Learningcited by 271
  12. 2017Meta Networkscited by 268
25 more
  1. 2018FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluationcited by 254
  2. 2019MMTM: Multimodal Transfer Module for CNN Fusioncited by 128
  3. 2021Multimodal Contrastive Training for Visual Representation Learningcited by 122
  4. 2020Deep Multimodal Fusion by Channel Exchangingcited by 117
  5. 2023Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Modelscited by 91
  6. 2018CentralNet: a Multilayer Approach for Multimodal Fusioncited by 86
  7. 2019BLOCK: Bilinear Superdiagonal Fusion for Visual Question Answering and Visual Relationship Detectioncited by 62
  8. 2020Few-shot Relation Extraction via Bayesian Meta-learning on Relation Graphscited by 39
  9. 2019Multi-Level Matching and Aggregation Network for Few-Shot Relation Classificationcited by 32
  10. 2021Exploring Task Difficulty for Few-Shot Relation Extractioncited by 24
  11. 2019Neural Snowball for Few-Shot Relation Learningcited by 22
  12. 2022Continual Few-shot Relation Learning via Embedding Space Regularization and Data Augmentationcited by 17
  13. 2023MEWL: Few-shot multimodal word learning with referential uncertaintycited by 17
  14. 2021MapRE: An Effective Semantic Mapping Approach for Low-resource Relation Extractioncited by 16
  15. 2021Entity Concept-enhanced Few-shot Relation Extractioncited by 14
  16. 2022Joint Multimodal Entity-Relation Extraction Based on Edge-enhanced Graph Alignment Network and Word-pair Relation Taggingcited by 14
  17. 2022A Simple yet Effective Relation Information Guided Approach for Few-Shot Relation Extractioncited by 9
  18. 2022Few-shot Multimodal Sentiment Analysis based on Multimodal Probabilistic Fusion Promptscited by 9
  19. 2020Zero-shot Relation Classification from Side Informationcited by 8
  20. 2021Prompt-based Zero-shot Relation Extraction with Semantic Knowledge Augmentationcited by 8
  21. 2022Better Few-Shot Relation Extraction with Label Prompt Dropoutcited by 8
  22. 2020Learning to Decouple Relations: Few-Shot Relation Classification with Entity-Guided Attention and Confusion-Aware Trainingcited by 7
  23. 2023Multimodal Relation Extraction with Cross-Modal Retrieval and Synthesiscited by 7
  24. 2023Dual-Gated Fusion with Prefix-Tuning for Multi-Modal Relation Extractioncited by 7
  25. 2022Function-words Enhanced Attention Networks for Few-Shot Inverse Relation Classificationcited by 5

Built on by

Not cited yet.