mlsanityData to 14 Oct 2026

BARThez: a Skilled Pretrained French Sequence-to-Sequence Model

arXiv 2010.12321 ↗First posted 23 Oct 2020cs.CLcited by 40

Not on the map. Why?

Abstract (arXiv)

Inductive transfer learning has taken the entire NLP field by storm, with models such as BERT and BART setting new state of the art on countless NLU tasks. However, most of the available models and research have been conducted for English. In this work, we introduce BARThez, the first large-scale pretrained seq2seq model for French. Being based on BART, BARThez is particularly well-suited for generative tasks. We evaluate BARThez on five discriminative tasks from the FLUE benchmark and two generative tasks from a novel summarization dataset, OrangeSum, that we created for this research. We show BARThez to be very competitive with state-of-the-art BERT-based French language models such as CamemBERT and FlauBERT. We also continue the pretraining of a multilingual BART on BARThez' corpus, and show our resulting model, mBARThez, to significantly boost BARThez' generative performance. Code, data and models are publicly available.

Key facts. Cited by 40 papers in the map (data through 14 Oct 2026). First posted on arXiv 23 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 8; 2022: 7; 2023: 9; 2024: 8; 2025: 5; 2026: 3.

40 papers.

Sort:

By area:

  1. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 360
  2. 2023ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learningcited by 206
  3. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 190
  4. 2021Language Models are Few-shot Multilingual Learnerscited by 92
  5. 2024CroissantLLM: A Truly Bilingual French-English Language Modelcited by 53
  6. 2022GEMv2: Multilingual NLG Benchmarking in a Single Line of Codecited by 36
  7. 2021BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamesecited by 34
  8. 2022AraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarizationcited by 24
  9. 2022Cedille: A large autoregressive French language modelcited by 20
  10. 2021FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metricsfor Automatic Text Generationcited by 19
  11. 2021JuriBERT: A Masked-Language Model Adaptation for French Legal Textcited by 14
  12. 2021BERTweetFR : Domain Adaptation of Pre-Trained Language Models for French Tweetscited by 9
28 more
  1. 2021PAGnol: An Extra-Large French Generative Modelcited by 9
  2. 2022Abstractive Meeting Summarization: A Surveycited by 9
  3. 2023GreekBART: The First Pretrained Greek Sequence-to-Sequence Modelcited by 8
  4. 2023Data-Efficient French Language Modeling with CamemBERTacited by 8
  5. 2025SENSE models: an open source solution for multilingual and multimodal semantic-based taskscited by 6
  6. 2024From News to Summaries: Building a Hungarian Corpus for Extractive and Abstractive Summarizationcited by 4
  7. 2021Evaluation Of Word Embeddings From Large-Scale French Web Contentcited by 3
  8. 2022Effectiveness of French Language Models on Abstractive Dialogue Summarization Taskcited by 3
  9. 2023Measuring Normative and Descriptive Biases in Language Models Using Census Datacited by 3
  10. 2023FREDSum: A Dialogue Summarization Corpus for French Political Debatescited by 3
  11. 2024The State and Fate of Summarization Datasets: A Surveycited by 3
  12. 2024RoLargeSum: A Large Dialect-Aware Romanian News Dataset for Summary, Headline, and Keyword Generationcited by 3
  13. 2023Safer Together: Machine Learning Models Trained on Shared Accident Datasets Predict Construction Injuries Better than Company-Specific Modelscited by 2
  14. 2023Sequence-to-Sequence Spanish Pre-trained Language Modelscited by 2
  15. 2024Increasing faithfulness in human-human dialog summarization with Spoken Language Understanding taskscited by 2
  16. 2025Facilitating Cognitive Accessibility with LLMs: A Multi-Task Approach to Easy-to-Read Text Generationcited by 2
  17. 2026Pantagruel: Unified Self-Supervised Encoders for French Text and Speechcited by 2
  18. 2022Social Media Sentiment Analysis for Cryptocurrency Market Predictioncited by 1
  19. 2024Know When to Fuse: Investigating Non-English Hybrid Retrieval in the Legal Domaincited by 1
  20. 2025Inclusive Easy-to-Read Generation for Individuals with Cognitive Impairmentscited by 1
  21. 2025AraFinNews: Arabic Financial Summarisation with Domain-Adapted LLMscited by 1
  22. 2025PrahokBART: A Pre-trained Sequence-to-Sequence Model for Khmer Natural Language Generationcited by 1
  23. 2021NLP Research and Resources at DaSciM, Ecole Polytechniquecited by 0
  24. 2022Word Sense Induction with Hierarchical Clustering and Mutual Information Maximizationcited by 0
  25. 2023LC-Score: Reference-less estimation of Text Comprehension Difficultycited by 0
  26. 2024Retrieve, Generate, Evaluate: A Case Study for Medical Paraphrases Generation with Small Language Modelscited by 0
  27. 2026Using Multimodal and Language-Agnostic Sentence Embeddings for Abstractive Summarizationcited by 0
  28. 2026Language Model Fingerprinting Requires Rethinking Watermark Teacherscited by 0