# CamemBERT 2.0: A Smarter French Language Model Aged to Perfection

arXiv: https://arxiv.org/abs/2411.08868 · HTML: https://mlsanity.com/paper/2411.08868/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Nov 2024.

## Built on by (most cited first, 14 of 14)

- [Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference](https://mlsanity.com/paper/2412.13663/) (2024)
- [ModernBERT or DeBERTaV3? Examining Architecture and Data Influence on Transformer Encoder Models Performance](https://mlsanity.com/paper/2504.08716/) (2025)
- [New Encoders for German Trained from Scratch: Comparing ModernGBERT with Converted LLM2Vec Models](https://mlsanity.com/paper/2505.13136/) (2025)
- [Zero-shot Large Language Models for Automatic Readability Assessment](https://mlsanity.com/paper/2604.24470/) (2026)
- [Pantagruel: Unified Self-Supervised Encoders for French Text and Speech](https://mlsanity.com/paper/2601.05911/) (2026)
- [Where Experts Disagree, Models Fail: Detecting Implicit Legal Citations in French Court Decisions](https://mlsanity.com/paper/2603.22973/) (2026)
- [JUDGEBERT: Assessing Legal Meaning Preservation Between Sentences](https://mlsanity.com/paper/2508.16870/) (2025)
- [Addressing Antisocial Behavior in Multi-Party Dialogs Through Multimodal Representation Learning](https://mlsanity.com/paper/2510.17289/) (2025)
- [The GDN-CC Dataset: Automatic Corpus Clarification for AI-enhanced Democratic Citizen Consultations](https://mlsanity.com/paper/2601.14944/) (2026)
- [Pretraining and Benchmarking Modern Encoders for Latvian](https://mlsanity.com/paper/2603.15005/) (2026)
- [Politics of Questions in News: A Mixed-Methods Study of Interrogative Stances as Markers of Voice and Power](https://mlsanity.com/paper/2603.21823/) (2026)
- [Model in Distress: Sentiment Analysis on French Synthetic Social Media](https://mlsanity.com/paper/2604.18226/) (2026)
- [moBERTo: A Modern Encoder for Portuguese via Continued Pretraining of ModernBERT](https://mlsanity.com/paper/2606.22722/) (2026)
- [FrenchNews-7: Benchmarking Cross-Publisher French News Editorial Desk Classification](https://mlsanity.com/paper/2608.18097/) (2026)

## Abstract (arXiv)

French language models, such as CamemBERT, have been widely adopted across industries for natural language processing (NLP) tasks, with models like CamemBERT seeing over 4 million downloads per month. However, these models face challenges due to temporal concept drift, where outdated training data leads to a decline in performance, especially when encountering new topics and terminology. This issue emphasizes the need for updated models that reflect current linguistic trends. In this paper, we introduce two new versions of the CamemBERT base model-CamemBERTav2 and CamemBERTv2-designed to address these challenges. CamemBERTav2 is based on the DeBERTaV3 architecture and makes use of the Replaced Token Detection (RTD) objective for better contextual understanding, while CamemBERTv2 is built on RoBERTa, which uses the Masked Language Modeling (MLM) objective. Both models are trained on a significantly larger and more recent dataset with longer context length and an updated tokenizer that enhances tokenization performance for French. We evaluate the performance of these models on both general-domain NLP tasks and domain-specific applications, such as medical field tasks, demonstrating their versatility and effectiveness across a range of use cases. Our results show that these updated models vastly outperform their predecessors, making them valuable tools for modern NLP systems. All our new models, as well as intermediate checkpoints, are made openly available on Huggingface.