# Learning Word Vectors for 157 Languages

arXiv: https://arxiv.org/abs/1802.06893 · HTML: https://mlsanity.com/paper/1802.06893/

## Key facts

- Cited by 280 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Feb 2018.

## Built on by (most cited first, 30 of 280)

- [The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only](https://mlsanity.com/paper/2306.01116/) (2023)
- [A critical look at the evaluation of GNNs under heterophily: Are we really making progress?](https://mlsanity.com/paper/2302.11640/) (2023)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages](https://mlsanity.com/paper/2309.09400/) (2023)
- [Towards a Cleaner Document-Oriented Multilingual Crawled Corpus](https://mlsanity.com/paper/2201.06642/) (2022)
- [XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond](https://mlsanity.com/paper/2104.12250/) (2021)
- [Big-Math: A Large-Scale, High-Quality Math Dataset for Reinforcement Learning in Language Models](https://mlsanity.com/paper/2502.17387/) (2025)
- [Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning](https://mlsanity.com/paper/2112.02706/) (2021)
- [What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus](https://mlsanity.com/paper/2105.02732/) (2021)
- [A Survey of Multimodal Large Language Model from A Data-centric Perspective](https://mlsanity.com/paper/2405.16640/) (2024)
- [MathPile: A Billion-Token-Scale Pretraining Corpus for Math](https://mlsanity.com/paper/2312.17120/) (2023)
- [Meta CLIP 2: A Worldwide Scaling Recipe](https://mlsanity.com/paper/2507.22062/) (2025)
- [Are Multilingual Models Effective in Code-Switching?](https://mlsanity.com/paper/2103.13309/) (2021)
- [ALLaM: Large Language Models for Arabic and English](https://mlsanity.com/paper/2407.15390/) (2024)
- [How to Adapt Your Pretrained Multilingual Model to 1600 Languages](https://mlsanity.com/paper/2106.02124/) (2021)
- [AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages](https://mlsanity.com/paper/2104.08726/) (2021)
- [Automatic Data Curation for Self-Supervised Learning: A Clustering-Based Approach](https://mlsanity.com/paper/2405.15613/) (2024)
- [Bitext Mining Using Distilled Sentence Representations for Low-Resource Languages](https://mlsanity.com/paper/2205.12654/) (2022)
- [Adapting BERT for Continual Learning of a Sequence of Aspect Sentiment Classification Tasks](https://mlsanity.com/paper/2112.03271/) (2021)
- [HateMM: A Multi-Modal Dataset for Hate Video Classification](https://mlsanity.com/paper/2305.03915/) (2023)
- [Large-Scale Hate Speech Detection with Cross-Domain Transfer](https://mlsanity.com/paper/2203.01111/) (2022)
- [A Statutory Article Retrieval Dataset in French](https://mlsanity.com/paper/2108.11792/) (2021)
- [Aioli: A Unified Optimization Framework for Language Model Data Mixing](https://mlsanity.com/paper/2411.05735/) (2024)
- [Large Language Models are Efficient Learners of Noise-Robust Speech Recognition](https://mlsanity.com/paper/2401.10446/) (2024)
- [CLASSIC: Continual and Contrastive Learning of Aspect Sentiment Classification Tasks](https://mlsanity.com/paper/2112.02714/) (2021)
- [A Large-scale Dataset for Hate Speech Detection on Vietnamese Social Media Texts](https://mlsanity.com/paper/2103.11528/) (2021)
- [Semantic Diversity Learning for Zero-Shot Multi-label Classification](https://mlsanity.com/paper/2105.05926/) (2021)
- [Better Feature Integration for Named Entity Recognition](https://mlsanity.com/paper/2104.05316/) (2021)
- [How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?](https://mlsanity.com/paper/2406.11477/) (2024)
- [Dataset for Identification of Homophobia and Transophobia in Multilingual YouTube Comments](https://mlsanity.com/paper/2109.00227/) (2021)

## Abstract (arXiv)

Distributed word representations, or word vectors, have recently been applied to many tasks in natural language processing, leading to state-of-the-art performance. A key ingredient to the successful application of these representations is to train them on very large corpora, and use these pre-trained models in downstream tasks. In this paper, we describe how we trained such high quality word representations for 157 languages. We used two sources of data to train these models: the free online encyclopedia Wikipedia and data from the common crawl project. We also introduce three new word analogy datasets to evaluate these word vectors, for French, Hindi and Polish. Finally, we evaluate our pre-trained word vectors on 10 languages for which evaluation datasets exists, showing very strong performance compared to previous models.