# 2M-NER: Contrastive Learning for Multilingual and Multimodal NER with Language and Modal Fusion

arXiv: https://arxiv.org/abs/2404.17122 · HTML: https://mlsanity.com/paper/2404.17122/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 59 references that are in the map.
- First posted on arXiv 26 Apr 2024.

## Builds on (most cited first, 25 of 25)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF](https://mlsanity.com/paper/1603.01354/) (2016)
- [Align and Prompt: Video-and-Language Pre-training with Entity Prompts](https://mlsanity.com/paper/2112.09583/) (2021)
- [Mintaka: A Complex, Natural, and Multilingual Dataset for End-to-End Question Answering](https://mlsanity.com/paper/2210.01613/) (2022)
- [Looking for a Needle in a Haystack: A Comprehensive Study of Hallucinations in Neural Machine Translation](https://mlsanity.com/paper/2208.05309/) (2022)
- [Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion](https://mlsanity.com/paper/2205.02357/) (2022)
- [Multimodal Named Entity Recognition for Short Social Media Posts](https://mlsanity.com/paper/1802.07862/) (2018)
- [RpBERT: A Text-image Relation Propagation-based BERT Model for Multimodal NER](https://mlsanity.com/paper/2102.02967/) (2021)
- [Enhanced Meta-Learning for Cross-lingual Named Entity Recognition with Minimal Resources](https://mlsanity.com/paper/1911.06161/) (2019)
- [ITA: Image-Text Alignments for Multi-Modal Named Entity Recognition](https://mlsanity.com/paper/2112.06482/) (2021)
- [Parallel Instance Query Network for Named Entity Recognition](https://mlsanity.com/paper/2203.10545/) (2022)
- [MultiCoNER: A Large-scale Multilingual dataset for Complex Named Entity Recognition](https://mlsanity.com/paper/2208.14536/) (2022)
- [DiS-ReX: A Multilingual Dataset for Distantly Supervised Relation Extraction](https://mlsanity.com/paper/2104.08655/) (2021)
- [Target-Oriented Fine-tuning for Zero-Resource Named Entity Recognition](https://mlsanity.com/paper/2107.10523/) (2021)
- [PARE: A Simple and Strong Baseline for Monolingual and Multilingual Distantly Supervised Relation Extraction](https://mlsanity.com/paper/2110.07415/) (2021)
- [HanoiT: Enhancing Context-aware Translation via Selective Context](https://mlsanity.com/paper/2301.06825/) (2023)
- [Multilingual Named Entity Recognition Using Pretrained Embeddings, Attention Mechanism and NCRF](https://mlsanity.com/paper/1906.09978/) (2019)

## Built on by (most cited first, 1 of 1)

- [MERLIN: A Testbed for Multilingual Multimodal Entity Recognition and Linking](https://mlsanity.com/paper/2510.14307/) (2025)

## Abstract (arXiv)

Named entity recognition (NER) is a fundamental task in natural language processing that involves identifying and classifying entities in sentences into pre-defined types. It plays a crucial role in various research fields, including entity linking, question answering, and online product recommendation. Recent studies have shown that incorporating multilingual and multimodal datasets can enhance the effectiveness of NER. This is due to language transfer learning and the presence of shared implicit features across different modalities. However, the lack of a dataset that combines multilingualism and multimodality has hindered research exploring the combination of these two aspects, as multimodality can help NER in multiple languages simultaneously. In this paper, we aim to address a more challenging task: multilingual and multimodal named entity recognition (MMNER), considering its potential value and influence. Specifically, we construct a large-scale MMNER dataset with four languages (English, French, German and Spanish) and two modalities (text and image). To tackle this challenging MMNER task on the dataset, we introduce a new model called 2M-NER, which aligns the text and image representations using contrastive learning and integrates a multimodal collaboration module to effectively depict the interactions between the two modalities. Extensive experimental results demonstrate that our model achieves the highest F1 score in multilingual and multimodal NER tasks compared to some comparative and representative baselines. Additionally, in a challenging analysis, we discovered that sentence-level alignment interferes a lot with NER models, indicating the higher level of difficulty in our dataset.