# A Multilingual Bag-of-Entities Model for Zero-Shot Cross-Lingual Text Classification

arXiv: https://arxiv.org/abs/2110.07792 · HTML: https://mlsanity.com/paper/2110.07792/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 40 references that are in the map.
- First posted on arXiv 15 Oct 2021.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Complex Embeddings for Simple Link Prediction](https://mlsanity.com/paper/1606.06357/) (2016)
- [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation](https://mlsanity.com/paper/1911.06136/) (2019)
- [LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention](https://mlsanity.com/paper/2010.01057/) (2020)
- [Knowledge Enhanced Contextual Word Representations](https://mlsanity.com/paper/1909.04164/) (2019)
- [Emerging Cross-lingual Structure in Pretrained Language Models](https://mlsanity.com/paper/1911.01464/) (2019)
- [Entities as Experts: Sparse Memory Access with Entity Supervision](https://mlsanity.com/paper/2004.07202/) (2020)
- [E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT](https://mlsanity.com/paper/1911.03681/) (2019)
- [A Corpus for Multilingual Document Classification in Eight Languages](https://mlsanity.com/paper/1805.09821/) (2018)
- [MultiFiT: Efficient Multi-lingual Language Model Fine-tuning](https://mlsanity.com/paper/1909.04761/) (2019)
- [Adversarial Learning with Contextual Embeddings for Zero-resource Cross-lingual Classification and NER](https://mlsanity.com/paper/1909.00153/) (2019)
- [Enriching BERT with Knowledge Graph Embeddings for Document Classification](https://mlsanity.com/paper/1909.08402/) (2019)
- [Multilingual Models for Compositional Distributed Semantics](https://mlsanity.com/paper/1404.4641/) (2014)
- [Injecting Knowledge Base Information into End-to-End Joint Entity and Relation Extraction and Coreference Resolution](https://mlsanity.com/paper/2107.02286/) (2021)
- [Leveraging Adversarial Training in Self-Learning for Cross-Lingual Text Classification](https://mlsanity.com/paper/2007.15072/) (2020)
- [Neural Attentive Bag-of-Entities Model for Text Classification](https://mlsanity.com/paper/1909.01259/) (2019)
- [Bridging the domain gap in cross-lingual document classification](https://mlsanity.com/paper/1909.07009/) (2019)

## Built on by (most cited first, 2 of 2)

- [EASE: Entity-Aware Contrastive Learning of Sentence Embedding](https://mlsanity.com/paper/2205.04260/) (2022)
- [Improving Zero-Shot Cross-Lingual Transfer via Progressive Code-Switching](https://mlsanity.com/paper/2406.13361/) (2024)

## Abstract (arXiv)

We present a multilingual bag-of-entities model that effectively boosts the performance of zero-shot cross-lingual text classification by extending a multilingual pre-trained language model (e.g., M-BERT). It leverages the multilingual nature of Wikidata: entities in multiple languages representing the same concept are defined with a unique identifier. This enables entities described in multiple languages to be represented using shared embeddings. A model trained on entity features in a resource-rich language can thus be directly applied to other languages. Our experimental results on cross-lingual topic classification (using the MLDoc and TED-CLDC datasets) and entity typing (using the SHINRA2020-ML dataset) show that the proposed model consistently outperforms state-of-the-art models.