FrenchNews-7: Benchmarking Cross-Publisher French News Editorial Desk Classification
Not on the map. Why?
Abstract (arXiv)
We present FrenchNews-7, a cross-publisher France-based French-language news editorial desk classification benchmark combining a large multi-outlet corpus, a URL-derived seven-class taxonomy, and a fine-tuned CamemBERT classifier. Labels are assigned via a hybrid pipeline combining publisher URL slugs with LLM annotation for structurally ambiguous cases, audited through an inter-rater study (2 humans + 2 LLMs; pairwise kappa ≥ 0.766, human--human kappa = 0.806). We evaluate lexical, multilingual, and French-specific trained classifiers under both in-distribution and held-out-publisher settings, with additional comparison against zero-shot LLM baselines (GPT-OSS-120B, Mistral Small 3.2, Llama-3.3-70B) on the held-out pool. The strongest model, CamemBERT-base on full article text, outperforms headline-only input, generalizes to unseen outlets, and exceeds all three zero-shot LLM baselines on overall recall (0.799), with the gap concentrated in the ambiguous editorial-boundary categories Economie and Societe. Cross-publisher evaluation reveals uneven boundary stability: Sport, Culture & Loisirs, and International transfer cleanly, while Economie (recall = 0.517) is close to blinded human agreement (0.55), and Societe (precision = 0.577) absorbs boundary ambiguity, both suggesting editorial conventions rather than recoverable classifier headroom. The fine-tuned CamemBERT-base model, labeled manifest, reference collection scripts, and a reliability-tier guidance table are available at https://huggingface.co/LeFrenchNewsLab/camembert-base-frenchnews7 (model) and https://huggingface.co/datasets/LeFrenchNewsLab/frenchnews-7 (dataset).
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 23 references that are in the map. First posted on arXiv 8 Jun 2026.
Builds on
16 of 23 references.
References · 16
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2017Axiomatic Attribution for Deep Networkscited by 3,384
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
- 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
- 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 870
- 2019CamemBERT: a Tasty French Language Modelcited by 368
- 2022Is GPT-3 a Good Data Annotator?cited by 208
- 2019FlauBERT: Unsupervised Language Model Pre-training for Frenchcited by 153
- 2020MLSUM: The Multilingual Summarization Corpuscited by 121
- 2018Framing and Agenda-setting in Russian News: a Computational Analysis of Intricate Political Strategiescited by 47
- 2024Knowledge Distillation in Automated Annotation: Supervised Text Classification with LLM-Generated Training Labelscited by 25
4 more
- 2024CamemBERT 2.0: A Smarter French Language Model Aged to Perfectioncited by 14
- 2024LLM Teacher-Student Framework for Text Classification With No Manually Annotated Data: A Case Study in IPTC News Topic Classificationcited by 6
- 2024Automatic Classification of News Subjects in Broadcast News: Application to a Gender Bias Representation Analysiscited by 2
- 2025Feeding LLM Annotations to BERT Classifiers at Your Own Riskcited by 2
Built on by
Not cited yet.