mlsanityData to 7 Oct 2026

RpBERT: A Text-image Relation Propagation-based BERT Model for Multimodal NER

arXiv 2102.02967 ↗First posted 5 Feb 2021cs.CL cs.LGcited by 37

Not on the map. Why?

Abstract (arXiv)

Recently multimodal named entity recognition (MNER) has utilized images to improve the accuracy of NER in tweets. However, most of the multimodal methods use attention mechanisms to extract visual clues regardless of whether the text and image are relevant. Practically, the irrelevant text-image pairs account for a large proportion in tweets. The visual clues that are unrelated to the texts will exert uncertain or even negative effects on multimodal model learning. In this paper, we introduce a method of text-image relation propagation into the multimodal BERT model. We integrate soft or hard gates to select visual clues and propose a multitask algorithm to train on the MNER datasets. In the experiments, we deeply analyze the changes in visual attention before and after the use of text-image relation propagation. Our model achieves state-of-the-art performance on the MNER datasets.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 20 of its 30 references that are in the map. First posted on arXiv 5 Feb 2021.

Builds on

20 of 30 references.

References · 20

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2014Going Deeper with Convolutionscited by 4,231
  5. 2015Fast R-CNNcited by 2,930
  6. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  10. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  11. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  12. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
8 more
  1. 2014Show and Tell: A Neural Image Caption Generatorcited by 877
  2. 2016Neural Architectures for Named Entity Recognitioncited by 548
  3. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  4. 2017Advances in Pre-Training Distributed Word Representationscited by 207
  5. 2018Multimodal Named Entity Recognition for Short Social Media Postscited by 43
  6. 2019Aiding Intra-Text Representations with Visual Context for Multimodal Named Entity Recognitioncited by 15
  7. 2019Open Event Extraction from Online Text using a Generative Adversarial Networkcited by 10
  8. 2017Twitter100k: A Real-world Dataset for Weakly Supervised Cross-Media Retrievalcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 8; 2023: 16; 2024: 7; 2025: 3; 2026: 2.

37 papers.

Sort:

By area:

  1. 2022Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completioncited by 44
  2. 2022Named Entity and Relation Extraction with Multi-Modal Retrievalcited by 26
  3. 2021ITA: Image-Text Alignments for Multi-Modal Named Entity Recognitioncited by 24
  4. 2022Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysiscited by 21
  5. 2024UMIE: Unified Multimodal Information Extraction with Instruction Tuningcited by 17
  6. 2024Large Language Models Meet Text-Centric Multimodal Sentiment Analysis: A Surveycited by 16
  7. 2023Prompting ChatGPT in MNER: Enhanced Multimodal Named Entity Recognition with Auxiliary Refined Knowledgecited by 11
  8. 2023Enhancing Multimodal Entity and Relation Extraction with Variational Information Bottleneckcited by 9
  9. 2023AoM: Detecting Aspect-oriented Information for Multimodal Aspect-Based Sentiment Analysiscited by 9
  10. 2022Flat Multi-modal Interaction Transformer for Named Entity Recognitioncited by 8
  11. 2023Learning Implicit Entity-object Relations by Bidirectional Generative Alignment for Multimodal NERcited by 8
  12. 2022Good Visual Guidance Makes A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extractioncited by 7
25 more
  1. 2023Dual-Gated Fusion with Prefix-Tuning for Multi-Modal Relation Extractioncited by 7
  2. 2025Collaborative Multi-LoRA Experts with Achievement-based Multi-Tasks Loss for Unified Multimodal Information Extractioncited by 7
  3. 2023Understanding Social Media Cross-Modality Discourse in Linguistic Spacecited by 6
  4. 2023Multimodal Question Answering for Unified Information Extractioncited by 6
  5. 2023M2DF: Multi-grained Multi-curriculum Denoising Framework for Multimodal Aspect-based Sentiment Analysiscited by 6
  6. 2023Prompt Me Up: Unleashing the Power of Alignments for Multimodal Entity and Relation Extractioncited by 5
  7. 2026Not Just What's There: Enabling CLIP to Comprehend Negated Visual Descriptions Without Fine-tuningcited by 5
  8. 2023Hierarchical Aligned Multimodal Learning for NER on Tweet Postscited by 4
  9. 2023Integrating Large Pre-trained Models into Multimodal Named Entity Recognition with Evidential Fusioncited by 4
  10. 2022Multi-Granularity Cross-Modality Representation Learning for Named Entity Recognition on Social Mediacited by 2
  11. 2022FedCL: Federated Multi-Phase Curriculum Learning to Synchronously Correlate User Heterogeneitycited by 2
  12. 2023Enhancing OCR Performance through Post-OCR Models: Adopting Glyph Embedding for Improved Correctioncited by 2
  13. 2023Exchanging-based Multimodal Fusion with Transformercited by 2
  14. 2023Towards Better Multi-modal Keyphrase Generation via Visual Entity Enhancement and Multi-granularity Image Noise Filteringcited by 2
  15. 2024Shapley Value-based Contrastive Alignment for Multimodal Information Extractioncited by 2
  16. 2025Enhanced Multimodal Aspect-Based Sentiment Analysis by LLM-Generated Rationalescited by 2
  17. 2022Boosting Multi-Modal E-commerce Attribute Value Extraction via Unified Learning Scheme and Dynamic Range Minimizationcited by 1
  18. 2023Evaluating the Robustness to Instructions of Large Language Modelscited by 1
  19. 20242M-NER: Contrastive Learning for Multilingual and Multimodal NER with Language and Modal Fusioncited by 1
  20. 2024Advancing Grounded Multimodal Named Entity Recognition via LLM-Based Reformulation and Box-Based Segmentationcited by 1
  21. 2025A Benchmark Dataset and a Framework for Urdu Multimodal Named Entity Recognitioncited by 1
  22. 2023Enhanced E-Commerce Attribute Extraction: Innovating with Decorative Relation Correction and LLAMA 2.0-Based Annotationcited by 0
  23. 2024Utilizing Large Language Models for Event Deconstruction to Enhance Multimodal Aspect-Based Sentiment Analysiscited by 0
  24. 2024A Dual-Module Denoising Approach with Curriculum Learning for Enhancing Multimodal Aspect-Based Sentiment Analysiscited by 0
  25. 2026DiffImaginE: Imagine to Verify Entity Types with Diffusioncited by 0