# BARThez: a Skilled Pretrained French Sequence-to-Sequence Model

arXiv: https://arxiv.org/abs/2010.12321 · HTML: https://mlsanity.com/paper/2010.12321/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Oct 2020.

## Built on by (most cited first, 30 of 40)

- [Better & Faster Large Language Models via Multi-token Prediction](https://mlsanity.com/paper/2404.19737/) (2024)
- [ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning](https://mlsanity.com/paper/2304.05613/) (2023)
- [Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.16039/) (2023)
- [Language Models are Few-shot Multilingual Learners](https://mlsanity.com/paper/2109.07684/) (2021)
- [CroissantLLM: A Truly Bilingual French-English Language Model](https://mlsanity.com/paper/2402.00786/) (2024)
- [GEMv2: Multilingual NLG Benchmarking in a Single Line of Code](https://mlsanity.com/paper/2206.11249/) (2022)
- [BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese](https://mlsanity.com/paper/2109.09701/) (2021)
- [AraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarization](https://mlsanity.com/paper/2203.10945/) (2022)
- [Cedille: A large autoregressive French language model](https://mlsanity.com/paper/2202.03371/) (2022)
- [FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metricsfor Automatic Text Generation](https://mlsanity.com/paper/2110.08559/) (2021)
- [JuriBERT: A Masked-Language Model Adaptation for French Legal Text](https://mlsanity.com/paper/2110.01485/) (2021)
- [BERTweetFR : Domain Adaptation of Pre-Trained Language Models for French Tweets](https://mlsanity.com/paper/2109.10234/) (2021)
- [PAGnol: An Extra-Large French Generative Model](https://mlsanity.com/paper/2110.08554/) (2021)
- [Abstractive Meeting Summarization: A Survey](https://mlsanity.com/paper/2208.04163/) (2022)
- [GreekBART: The First Pretrained Greek Sequence-to-Sequence Model](https://mlsanity.com/paper/2304.00869/) (2023)
- [Data-Efficient French Language Modeling with CamemBERTa](https://mlsanity.com/paper/2306.01497/) (2023)
- [SENSE models: an open source solution for multilingual and multimodal semantic-based tasks](https://mlsanity.com/paper/2509.12093/) (2025)
- [From News to Summaries: Building a Hungarian Corpus for Extractive and Abstractive Summarization](https://mlsanity.com/paper/2404.03555/) (2024)
- [Evaluation Of Word Embeddings From Large-Scale French Web Content](https://mlsanity.com/paper/2105.01990/) (2021)
- [Effectiveness of French Language Models on Abstractive Dialogue Summarization Task](https://mlsanity.com/paper/2207.08305/) (2022)
- [Measuring Normative and Descriptive Biases in Language Models Using Census Data](https://mlsanity.com/paper/2304.05764/) (2023)
- [FREDSum: A Dialogue Summarization Corpus for French Political Debates](https://mlsanity.com/paper/2312.04843/) (2023)
- [The State and Fate of Summarization Datasets: A Survey](https://mlsanity.com/paper/2411.04585/) (2024)
- [RoLargeSum: A Large Dialect-Aware Romanian News Dataset for Summary, Headline, and Keyword Generation](https://mlsanity.com/paper/2412.11317/) (2024)
- [Safer Together: Machine Learning Models Trained on Shared Accident Datasets Predict Construction Injuries Better than Company-Specific Models](https://mlsanity.com/paper/2301.03567/) (2023)
- [Sequence-to-Sequence Spanish Pre-trained Language Models](https://mlsanity.com/paper/2309.11259/) (2023)
- [Increasing faithfulness in human-human dialog summarization with Spoken Language Understanding tasks](https://mlsanity.com/paper/2409.10070/) (2024)
- [Facilitating Cognitive Accessibility with LLMs: A Multi-Task Approach to Easy-to-Read Text Generation](https://mlsanity.com/paper/2510.00662/) (2025)
- [Pantagruel: Unified Self-Supervised Encoders for French Text and Speech](https://mlsanity.com/paper/2601.05911/) (2026)
- [Social Media Sentiment Analysis for Cryptocurrency Market Prediction](https://mlsanity.com/paper/2204.10185/) (2022)

## Abstract (arXiv)

Inductive transfer learning has taken the entire NLP field by storm, with models such as BERT and BART setting new state of the art on countless NLU tasks. However, most of the available models and research have been conducted for English. In this work, we introduce BARThez, the first large-scale pretrained seq2seq model for French. Being based on BART, BARThez is particularly well-suited for generative tasks. We evaluate BARThez on five discriminative tasks from the FLUE benchmark and two generative tasks from a novel summarization dataset, OrangeSum, that we created for this research. We show BARThez to be very competitive with state-of-the-art BERT-based French language models such as CamemBERT and FlauBERT. We also continue the pretraining of a multilingual BART on BARThez' corpus, and show our resulting model, mBARThez, to significantly boost BARThez' generative performance. Code, data and models are publicly available.