COLE: a Comprehensive Benchmark for French Language Understanding Evaluation
Not on the map. Why?
Abstract (arXiv)
To address the need for a more comprehensive evaluation of French Natural Language Understanding (NLU), we introduce COLE, a new benchmark composed of 23 diverse task covering a broad range of NLU capabilities, including sentiment analysis, paraphrase detection, grammatical judgment, and reasoning, with a particular focus on linguistic phenomena relevant to the French language. We benchmark 94 large language models (LLM), providing an extensive analysis of the current state of French NLU. Our results highlight a significant performance gap between closed- and open-weights models and identify key challenging frontiers for current LLMs, such as zero-shot extractive question-answering (QA), fine-grained word sense disambiguation, and understanding of regional language variations. We release COLE as a public resource to foster further progress in French language modelling.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 33 of its 83 references that are in the map. First posted on arXiv 6 Oct 2025.
Builds on
33 of 83 references.
References · 33
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
- 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
- 2020Knowledge Distillation: A Surveycited by 1,578
- 2018Datasheets for Datasetscited by 1,371
21 more
- 2023A Survey on Evaluation of Large Language Modelscited by 1,227
- 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
- 2018Neural Network Acceptability Judgmentscited by 952
- 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
- 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
- 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
- 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 307
- 2020CLUE: A Chinese Language Understanding Evaluation Benchmarkcited by 277
- 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
- 2019FlauBERT: Unsupervised Language Model Pre-training for Frenchcited by 153
- 2024Investigating Data Contamination for Pre-training Language Modelscited by 91
- 2018A Span-Extraction Dataset for Chinese Machine Reading Comprehensioncited by 84
- 2025MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairscited by 65
- 2020FQuAD: French Question Answering Datasetcited by 58
- 2018DRCD: a Chinese Machine Reading Comprehension Datasetcited by 57
- 2024CroissantLLM: A Truly Bilingual French-English Language Modelcited by 52
- 2021Does Putting a Linguist in the Loop Improve NLU Data Collection?cited by 29
- 2020Project PIAF: Building a Native French Question-Answering Datasetcited by 13
- 2022Generalized Quantifiers as a Source of Error in Multilingual NLU Benchmarkscited by 8
- 2023FRACAS: A FRench Annotated Corpus of Attribution relations in newScited by 3
- 2025QFrBLiMP: a Quebec-French Benchmark of Linguistic Minimal Pairscited by 2
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.