# Multi-Granularity Cross-Modality Representation Learning for Named Entity Recognition on Social Media

arXiv: https://arxiv.org/abs/2210.14163 · HTML: https://mlsanity.com/paper/2210.14163/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 41 references that are in the map.
- First posted on arXiv 19 Oct 2022.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [Named Entity Recognition with Bidirectional LSTM-CNNs](https://mlsanity.com/paper/1511.08308/) (2015)
- [Unified Named Entity Recognition as Word-Word Relation Classification](https://mlsanity.com/paper/2112.10070/) (2021)
- [A Closer Look At Feature Space Data Augmentation For Few-Shot Intent Classification](https://mlsanity.com/paper/1910.04176/) (2019)
- [Multimodal Named Entity Recognition for Short Social Media Posts](https://mlsanity.com/paper/1802.07862/) (2018)
- [Fast and Accurate Entity Recognition with Iterated Dilated Convolutions](https://mlsanity.com/paper/1702.02098/) (2017)
- [RpBERT: A Text-image Relation Propagation-based BERT Model for Multimodal NER](https://mlsanity.com/paper/2102.02967/) (2021)
- [Hierarchical Contextualized Representation for Named Entity Recognition](https://mlsanity.com/paper/1911.02257/) (2019)
- [Dependency-Guided LSTM-CRF for Named Entity Recognition](https://mlsanity.com/paper/1909.10148/) (2019)
- [Named Entity Recognition for Social Media Texts with Semantic Augmentation](https://mlsanity.com/paper/2010.15458/) (2020)
- [A Multi-task Approach for Named Entity Recognition in Social Media Data](https://mlsanity.com/paper/1906.04135/) (2019)
- [Lexicon Infused Phrase Embeddings for Named Entity Resolution](https://mlsanity.com/paper/1404.5367/) (2014)
- [Aiding Intra-Text Representations with Visual Context for Multimodal Named Entity Recognition](https://mlsanity.com/paper/1904.01356/) (2019)
- [Named Entity Recognition with stack residual LSTM and trainable bias decoding](https://mlsanity.com/paper/1706.07598/) (2017)

## Built on by (most cited first, 2 of 2)

- [Fine-Grained Distillation for Long Document Retrieval](https://mlsanity.com/paper/2212.10423/) (2022)
- [Hierarchical Aligned Multimodal Learning for NER on Tweet Posts](https://mlsanity.com/paper/2305.08372/) (2023)

## Abstract (arXiv)

Named Entity Recognition (NER) on social media refers to discovering and classifying entities from unstructured free-form content, and it plays an important role for various applications such as intention understanding and user recommendation. With social media posts tending to be multimodal, Multimodal Named Entity Recognition (MNER) for the text with its accompanying image is attracting more and more attention since some textual components can only be understood in combination with visual information. However, there are two drawbacks in existing approaches: 1) Meanings of the text and its accompanying image do not match always, so the text information still plays a major role. However, social media posts are usually shorter and more informal compared with other normal contents, which easily causes incomplete semantic description and the data sparsity problem. 2) Although the visual representations of whole images or objects are already used, existing methods ignore either fine-grained semantic correspondence between objects in images and words in text or the objective fact that there are misleading objects or no objects in some images. In this work, we solve the above two problems by introducing the multi-granularity cross-modality representation learning. To resolve the first problem, we enhance the representation by semantic augmentation for each word in text. As for the second issue, we perform the cross-modality semantic interaction between text and vision at the different vision granularity to get the most effective multimodal guidance representation for every word. Experiments show that our proposed approach can achieve the SOTA or approximate SOTA performance on two benchmark datasets of tweets. The code, data and the best performing models are available at https://github.com/LiuPeiP-CS/IIE4MNER