mlsanityData to 14 Oct 2026

Good Visual Guidance Makes A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction

arXiv 2205.03521 ↗First posted 7 May 2022cs.CL cs.AI cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

Multimodal named entity recognition and relation extraction (MNER and MRE) is a fundamental and crucial branch in information extraction. However, existing approaches for MNER and MRE usually suffer from error sensitivity when irrelevant object images incorporated in texts. To deal with these issues, we propose a novel Hierarchical Visual Prefix fusion NeTwork (HVPNeT) for visual-enhanced entity and relation extraction, aiming to achieve more effective and robust performance. Specifically, we regard visual representation as pluggable visual prefix to guide the textual representation for error insensitive forecasting decision. We further propose a dynamic gated aggregation strategy to achieve hierarchical multi-scaled visual features as visual prefix for fusion. Extensive experiments on three benchmark datasets demonstrate the effectiveness of our method, and achieve state-of-the-art performance. Code is available in https://github.com/zjunlp/HVPNeT.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 30 of its 47 references that are in the map. First posted on arXiv 7 May 2022.

Builds on

30 of 47 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  4. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
  5. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  7. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  8. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  9. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  10. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  11. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  12. 2018Path Aggregation Network for Instance Segmentationcited by 798
18 more
  1. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  2. 2016Neural Architectures for Named Entity Recognitioncited by 550
  3. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  4. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 356
  5. 2016End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRFcited by 341
  6. 2019Matching the Blanks: Distributional Similarity for Relation Learningcited by 272
  7. 2019A Fast and Accurate One-Stage Approach to Visual Groundingcited by 195
  8. 2021KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extractioncited by 180
  9. 2021Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learnerscited by 115
  10. 2021Document-level Relation Extraction as Semantic Segmentationcited by 80
  11. 2020ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learningcited by 72
  12. 2018Multimodal Named Entity Recognition for Short Social Media Postscited by 43
  13. 2021RpBERT: A Text-image Relation Propagation-based BERT Model for Multimodal NERcited by 37
  14. 2021AliCG: Fine-grained and Evolvable Conceptual Graph Construction for Semantic Search at Alibabacited by 33
  15. 2021Noisy-Labeled NER with Confidence Estimationcited by 28
  16. 2019GCDT: A Global Context Enhanced Deep Transition Architecture for Sequence Labelingcited by 22
  17. 2019Aiding Intra-Text Representations with Visual Context for Multimodal Named Entity Recognitioncited by 15
  18. 2022Contrastive Demonstration Tuning for Pre-trained Language Modelscited by 7

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 5; 2024: 0; 2025: 1.

7 papers.

  1. 2023Visual Tuningcited by 34
  2. 2023Information Screening whilst Exploiting! Multimodal Relation Extraction with Feature Denoising and Multimodal Topic Modelingcited by 16
  3. 2022Towards Relation Extraction From Speechcited by 10
  4. 2023Learning Implicit Entity-object Relations by Bidirectional Generative Alignment for Multimodal NERcited by 8
  5. 2023Integrating Large Pre-trained Models into Multimodal Named Entity Recognition with Evidential Fusioncited by 4
  6. 2023I2SRM: Intra- and Inter-Sample Relationship Modeling for Multimodal Information Extractioncited by 1
  7. 2025A Benchmark Dataset and a Framework for Urdu Multimodal Named Entity Recognitioncited by 1