mlsanityData to 7 Oct 2026

Multimodal Named Entity Recognition for Short Social Media Posts

arXiv 1802.07862 ↗First posted 22 Feb 2018cs.CLcited by 43

Not on the map. Why?

Abstract (arXiv)

We introduce a new task called Multimodal Named Entity Recognition (MNER) for noisy user-generated data such as tweets or Snapchat captions, which comprise short text with accompanying images. These social media posts often come in inconsistent or incomplete syntax and lexical notations with very limited surrounding textual contexts, bringing significant challenges for NER. To this end, we create a new dataset for MNER called SnapCaptions (Snapchat image-caption pairs submitted to public and crowd-sourced stories with fully annotated named entities). We then build upon the state-of-the-art Bi-LSTM word/character based NER models with 1) a deep image network which incorporates relevant visual context to augment textual information, and 2) a generic modality-attention module which learns to attenuate irrelevant modalities while amplifying the most informative ones to extract contexts from, adaptive to each sample and token. The proposed MNER model with modality attention significantly outperforms the state-of-the-art text-only NER models by successfully leveraging provided visual contexts, opening up potential applications of MNER on myriads of social media platforms.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 8; 2022: 9; 2023: 14; 2024: 9; 2025: 0; 2026: 3.

43 papers.

Sort:

By area:

  1. 2021Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysiscited by 45
  2. 2022Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completioncited by 44
  3. 2021RpBERT: A Text-image Relation Propagation-based BERT Model for Multimodal NERcited by 37
  4. 2021A Survey on Multi-modal Summarizationcited by 26
  5. 2022Named Entity and Relation Extraction with Multi-Modal Retrievalcited by 26
  6. 2021ITA: Image-Text Alignments for Multi-Modal Named Entity Recognitioncited by 24
  7. 2023NBIAS: A Natural Language Processing Framework for Bias Identification in Textcited by 23
  8. 2024UMIE: Unified Multimodal Information Extraction with Instruction Tuningcited by 17
  9. 2022MNER-QG: An End-to-End MRC framework for Multimodal Named Entity Recognition with Query Groundingcited by 16
  10. 2024Large Language Models Meet Text-Centric Multimodal Sentiment Analysis: A Surveycited by 16
  11. 2023Continual Multimodal Knowledge Graph Constructioncited by 15
  12. 2022Joint Multimodal Entity-Relation Extraction Based on Edge-enhanced Graph Alignment Network and Word-pair Relation Taggingcited by 14
31 more
  1. 2023Single-branch Network for Multimodal Trainingcited by 13
  2. 2023MESED: A Multi-modal Entity Set Expansion Dataset with Fine-grained Semantic Classes and Hard Negative Entitiescited by 13
  3. 2023Construction of Knowledge Graphs: State and Challengescited by 12
  4. 2021Context-NER : Contextual Phrase Generation at Scalecited by 11
  5. 2023Prompting ChatGPT in MNER: Enhanced Multimodal Named Entity Recognition with Auxiliary Refined Knowledgecited by 11
  6. 2024Resolving Word Vagueness with Scenario-guided Adapter for Natural Language Inferencecited by 11
  7. 2021Point-of-Interest Type Prediction using Text and Imagescited by 10
  8. 2021A Differentiable Language Model Adversarial Attack on Text Classifierscited by 9
  9. 2023Enhancing Multimodal Entity and Relation Extraction with Variational Information Bottleneckcited by 9
  10. 2022Flat Multi-modal Interaction Transformer for Named Entity Recognitioncited by 8
  11. 2022Kuaipedia: a Large-scale Multi-modal Short-video Encyclopediacited by 8
  12. 2022Good Visual Guidance Makes A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extractioncited by 7
  13. 2024LLMs as Bridges: Reformulating Grounded Multimodal Named Entity Recognitioncited by 7
  14. 2023Multimodal Question Answering for Unified Information Extractioncited by 6
  15. 2023M2DF: Multi-grained Multi-curriculum Denoising Framework for Multimodal Aspect-based Sentiment Analysiscited by 6
  16. 2026Beyond Unimodal Shortcuts: MLLMs as Cross-Modal Reasoners for Grounded Named Entity Recognitioncited by 6
  17. 2023Integrating Large Pre-trained Models into Multimodal Named Entity Recognition with Evidential Fusioncited by 4
  18. 2024Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approachcited by 4
  19. 2021Automated PII Extraction from Social Media for Raising Privacy Awareness: A Deep Transfer Learning Approachcited by 3
  20. 2022Multi-Granularity Cross-Modality Representation Learning for Named Entity Recognition on Social Mediacited by 2
  21. 2023Exchanging-based Multimodal Fusion with Transformercited by 2
  22. 2023Improving Multimodal Classification of Social Media Posts by Leveraging Image-Text Auxiliary Taskscited by 2
  23. 2024CMNER: A Chinese Multimodal NER Dataset based on Social Mediacited by 2
  24. 2024Chameleon: Images Are What You Need For Multimodal Learning Robust To Missing Modalitiescited by 2
  25. 2022DM2S2: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attentioncited by 1
  26. 2023I2SRM: Intra- and Inter-Sample Relationship Modeling for Multimodal Information Extractioncited by 1
  27. 2023Well Begun is Half Done: Generator-agnostic Knowledge Pre-Selection for Knowledge-Grounded Dialoguecited by 1
  28. 20242M-NER: Contrastive Learning for Multilingual and Multimodal NER with Language and Modal Fusioncited by 1
  29. 2024Advancing Grounded Multimodal Named Entity Recognition via LLM-Based Reformulation and Box-Based Segmentationcited by 1
  30. 2026SAMA: Semantic Anchor-aligned Augmentation for Unified Low-Resource Multimodal Information Extractioncited by 0
  31. 2026DiffImaginE: Imagine to Verify Entity Types with Diffusioncited by 0