mlsanityData to 14 Oct 2026

Prompt Me Up: Unleashing the Power of Alignments for Multimodal Entity and Relation Extraction

arXiv 2310.16822 ↗First posted 25 Oct 2023cs.CL cs.AI cs.MMcited by 5

Not on the map. Why?

Abstract (arXiv)

How can we better extract entities and relations from text? Using multimodal extraction with images and text obtains more signals for entities and relations, and aligns them through graphs or hierarchical fusion, aiding in extraction. Despite attempts at various fusions, previous works have overlooked many unlabeled image-caption pairs, such as NewsCLIPing. This paper proposes innovative pre-training objectives for entity-object and relation-image alignment, extracting objects from images and aligning them with entity and relation prompts for soft pseudo-labels. These labels are used as self-supervised signals for pre-training, enhancing the ability to extract entities and relations. Experiments on three datasets show an average 3.41% F1 improvement over prior SOTA. Additionally, our method is orthogonal to previous multimodal fusions, and using it on prior SOTA fusions further improves 5.47% F1.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 34 of its 64 references that are in the map. First posted on arXiv 25 Oct 2023.

Builds on

34 of 64 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  5. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  6. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  7. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  8. 2019When Does Label Smoothing Help?cited by 988
  9. 2022YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectorscited by 982
  10. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  11. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  12. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 356
22 more
  1. 2016End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRFcited by 341
  2. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  3. 2019Matching the Blanks: Distributional Similarity for Relation Learningcited by 272
  4. 2023A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capabilitycited by 119
  5. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  6. 2020Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learningcited by 74
  7. 2022Video Transformers: A Surveycited by 63
  8. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  9. 2020SelfORE: Self-supervised Relational Feature Learning for Open Relation Extractioncited by 43
  10. 2021RpBERT: A Text-image Relation Propagation-based BERT Model for Multimodal NERcited by 37
  11. 2021Gradient Imitation Reinforcement Learning for Low Resource Relation Extractioncited by 33
  12. 2022HiURE: Hierarchical Exemplar Contrastive Learning for Unsupervised Relation Extractioncited by 32
  13. 2022Named Entity and Relation Extraction with Multi-Modal Retrievalcited by 26
  14. 2022Semantic Enhanced Text-to-SQL Parsing via Iteratively Learning Schema Linking Graphcited by 22
  15. 2022Multimodal Learning on Graphs for Disease Relation Extractioncited by 15
  16. 2023GDA: Generative Data Augmentation Techniques for Relation Extraction Taskscited by 10
  17. 2023Enhancing Multimodal Entity and Relation Extraction with Variational Information Bottleneckcited by 9
  18. 2022Entity-to-Text based Data Augmentation for various Named Entity Recognition Taskscited by 6
  19. 2023Think Rationally about What You See: Continuous Rationale Extraction for Relation Extractioncited by 6
  20. 2022On Analyzing the Role of Image for Visual-enhanced Relation Extractioncited by 5
  21. 2022Survey on Self-Supervised Multimodal Representation Learning and Foundation Modelscited by 2
  22. 2023Exploring the Compositional Generalization in Context Dependent Text-to-SQL Parsingcited by 2

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 3.

5 papers.

  1. 2024TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understandingcited by 38
  2. 2023Training on Synthetic Data Beats Real Data in Multimodal Relation Extractioncited by 2
  3. 2025REMOTE: A Unified Multimodal Relation Extraction Framework with Multilevel Optimal Transport and Mixture-of-Expertscited by 2
  4. 2025Dependency Structure Augmented Contextual Scoping Framework for Multimodal Aspect-Based Sentiment Analysiscited by 1
  5. 2025Dynamic Relation Inference via Verb Embeddingscited by 0