DistilCamemBERT: a distillation of the French model CamemBERT
Not on the map. Why?
Abstract (arXiv)
Modern Natural Language Processing (NLP) models based on Transformer structures represent the state of the art in terms of performance on very diverse tasks. However, these models are complex and represent several hundred million parameters for the smallest of them. This may hinder their adoption at the industrial level, making it difficult to scale up to a reasonable infrastructure and/or to comply with societal and environmental responsibilities. To this end, we present in this paper a model that drastically reduces the computational cost of a well-known French model (CamemBERT), while preserving good performance.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 10 of its 20 references that are in the map. First posted on arXiv 23 May 2022.
Builds on
10 of 20 references.
References · 10
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
- 2018Deep contextualized word representationscited by 2,285
- 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
- 2019CamemBERT: a Tasty French Language Modelcited by 368
- 2019FlauBERT: Unsupervised Language Model Pre-training for Frenchcited by 153
Built on by
Citing papers per year: 2024: 1.
1 paper.