mlsanityData to 14 Oct 2026

CamemBERT: a Tasty French Language Model

arXiv 1911.03894 ↗First posted 10 Nov 2019cs.CLcited by 368

Not on the map. Why?

Abstract (arXiv)

Pretrained language models are now ubiquitous in Natural Language Processing. Despite their success, most available models have either been trained on English data or on the concatenation of data in multiple languages. This makes practical use of such models --in all languages except English-- very limited. In this paper, we investigate the feasibility of training monolingual Transformer-based language models for other languages, taking French as an example and evaluating our language models on part-of-speech tagging, dependency parsing, named entity recognition and natural language inference tasks. We show that the use of web crawled data is preferable to the use of Wikipedia data. More surprisingly, we show that a relatively small web crawled dataset (4GB) leads to results that are as good as those obtained using larger datasets (130+GB). Our best performing model CamemBERT reaches or improves the state of the art in all four downstream tasks.

Key facts. Cited by 368 papers in the map (data through 14 Oct 2026). First posted on arXiv 10 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 72; 2022: 74; 2023: 84; 2024: 67; 2025: 36; 2026: 35.

368 papers.

Sort:

By area:

  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  2. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  3. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 243
  4. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
  5. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 206
  6. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 190
  7. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  8. 2022mGPT: Few-Shot Learners Go Multilingualcited by 142
  9. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  10. 2021BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Banglacited by 125
  11. 2021Language Models are Few-shot Multilingual Learnerscited by 92
  12. 2021MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfercited by 90
38 more
  1. 2021WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language modelscited by 89
  2. 2023Spanish Pre-trained BERT Model and Evaluation Datacited by 81
  3. 2021How to Train BERT with an Academic Budgetcited by 80
  4. 2022Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methodscited by 79
  5. 2021What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformerscited by 76
  6. 2023LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domaincited by 68
  7. 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63
  8. 2021Swiss-Judgment-Prediction: A Multilingual Legal Judgment Prediction Benchmarkcited by 60
  9. 2022Gender Bias in Masked Language Models for Multiple Languagescited by 60
  10. 2021IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generationcited by 57
  11. 2021HerBERT: Efficiently Pretrained Transformer-based Language Model for Polishcited by 51
  12. 2021A Primer on Pretrained Multilingual Language Modelscited by 49
  13. 2023Attention Scheme Inspired Softmax Regressioncited by 49
  14. 2021Named Entity Recognition and Classification on Historical Documents: A Surveycited by 47
  15. 2022Long Document Summarization with Top-down and Bottom-up Inferencecited by 42
  16. 2024Quantifying the Gain in Weak-to-Strong Generalizationcited by 41
  17. 2021A Statutory Article Retrieval Dataset in Frenchcited by 40
  18. 2021Pre-Training BERT on Arabic Tweets: Practical Considerationscited by 39
  19. 2023Trained on 100 million words and still in shape: BERT meets British National Corpuscited by 39
  20. 2023Randomized and Deterministic Attention Sparsification Algorithms for Over-parameterized Feature Dimensioncited by 37
  21. 2025Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biasescited by 37
  22. 2021MarIA: Spanish Language Modelscited by 36
  23. 2021BERT, mBERT, or BiBERT? A Study on Contextualized Embeddings for Neural Machine Translationcited by 35
  24. 2023An Iterative Algorithm for Rescaled Hyperbolic Functions Regressioncited by 35
  25. 2023Lost in Translation: Large Language Models in Non-English Content Analysiscited by 35
  26. 2023Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Modelscited by 35
  27. 2021RobeCzech: Czech RoBERTa, a monolingual contextualized language representation modelcited by 33
  28. 2021AMMU : A Survey of Transformer-based Biomedical Pretrained Language Modelscited by 32
  29. 2023Typhoon: Thai Large Language Modelscited by 30
  30. 2024Training Tensor Attention Efficiently: From Cubic to Almost Linear Timecited by 30
  31. 2023Dissociating language and thought in large language modelscited by 29
  32. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 29
  33. 2021A systematic review of Hate Speech automatic detection using Natural Language Processingcited by 27
  34. 2022KinyaBERT: a Morphology-aware Kinyarwanda Language Modelcited by 27
  35. 2023DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domainscited by 27
  36. 2023A Family of Pretrained Transformer Language Models for Russiancited by 27
  37. 2021WangchanBERTa: Pretraining transformer-based Thai Language Modelscited by 26
  38. 2021Are Multilingual Models the Best Choice for Moderately Under-resourced Languages? A Comprehensive Assessment for Catalancited by 26

50 most cited of 368 All 368 (JSON)