mlsanityData to 7 Oct 2026

BERTweetFR : Domain Adaptation of Pre-Trained Language Models for French Tweets

arXiv 2109.10234 ↗First posted 21 Sep 2021cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

We introduce BERTweetFR, the first large-scale pre-trained language model for French tweets. Our model is initialized using the general-domain French language model CamemBERT which follows the base architecture of RoBERTa. Experiments show that BERTweetFR outperforms all previous general-domain French language models on two downstream Twitter NLP tasks of offensiveness identification and named entity recognition. The dataset used in the offensiveness detection task is first created and annotated by our team, filling in the gap of such analytic datasets in French. We make our model publicly available in the transformers library with the aim of promoting future research in analytic tasks for French tweets.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 17 of its 24 references that are in the map. First posted on arXiv 21 Sep 2021.

Builds on

17 of 24 references.

References · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  7. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  8. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  9. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  10. 2019CamemBERT: a Tasty French Language Modelcited by 368
  11. 2020BERTweet: A pre-trained language model for English Tweetscited by 346
  12. 2015Semi-supervised Sequence Learningcited by 198
5 more
  1. 2019FlauBERT: Unsupervised Language Model Pre-training for Frenchcited by 153
  2. 2016Unsupervised Pretraining for Sequence to Sequence Learningcited by 47
  3. 2020BARThez: a Skilled Pretrained French Sequence-to-Sequence Modelcited by 39
  4. 2018Socioeconomic Dependencies of Linguistic Patterns in Twitter: A Multivariate Analysiscited by 9
  5. 2017CAp 2017 challenge: Twitter Named Entity Recognitioncited by 2

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 3; 2023: 2; 2024: 3.

9 papers.

  1. 2023GeoLM: Empowering Language Models for Geospatially Grounded Language Understandingcited by 37
  2. 2022On the Domain Adaptation and Generalization of Pretrained Language Models: A Surveycited by 22
  3. 2022Using Emotion Embeddings to Transfer Knowledge Between Emotions, Languages, and Annotation Formatscited by 7
  4. 2024DALLMi: Domain Adaption for LLM-based Multi-label Classifiercited by 7
  5. 2024Code-mixed Sentiment and Hate-speech Predictioncited by 4
  6. 2024BANER: Boundary-Aware LLMs for Few-Shot Named Entity Recognitioncited by 3
  7. 2022Political Communities on Twitter: Case Study of the 2022 French Presidential Electioncited by 1
  8. 2021NLP Research and Resources at DaSciM, Ecole Polytechniquecited by 0
  9. 2023RoBERTweet: A BERT Language Model for Romanian Tweetscited by 0