# Beyond the English Web: Zero-Shot Cross-Lingual and Lightweight Monolingual Classification of Registers

arXiv: https://arxiv.org/abs/2102.07396 · HTML: https://mlsanity.com/paper/2102.07396/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 33 references that are in the map.
- First posted on arXiv 15 Feb 2021.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data](https://mlsanity.com/paper/1911.00359/) (2019)
- [Word Translation Without Parallel Data](https://mlsanity.com/paper/1710.04087/) (2017)
- [Multilingual is not enough: BERT for Finnish](https://mlsanity.com/paper/1912.07076/) (2019)
- [On the Language Neutrality of Pre-trained Multilingual Representations](https://mlsanity.com/paper/2004.05160/) (2020)
- [Playing with Words at the National Library of Sweden -- Making a Swedish BERT](https://mlsanity.com/paper/2007.01658/) (2020)
- [UPB at SemEval-2020 Task 12: Multilingual Offensive Language Detection on Social Media by Fine-tuning a Variety of BERT-based Models](https://mlsanity.com/paper/2010.13609/) (2020)

## Built on by (most cited first, 5 of 5)

- [ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification](https://mlsanity.com/paper/2303.03953/) (2023)
- [The GINCO Training Dataset for Web Genre Identification of Documents Out in the Wild](https://mlsanity.com/paper/2201.03857/) (2022)
- [Register Always Matters: Analysis of LLM Pretraining Data Through the Lens of Language Variation](https://mlsanity.com/paper/2504.01542/) (2025)
- [Automatic register identification for the open web using multilingual deep learning](https://mlsanity.com/paper/2406.19892/) (2024)
- [Explaining Classes through Word Attribution](https://mlsanity.com/paper/2108.13653/) (2021)

## Abstract (arXiv)

We explore cross-lingual transfer of register classification for web documents. Registers, that is, text varieties such as blogs or news are one of the primary predictors of linguistic variation and thus affect the automatic processing of language. We introduce two new register annotated corpora, FreCORE and SweCORE, for French and Swedish. We demonstrate that deep pre-trained language models perform strongly in these languages and outperform previous state-of-the-art in English and Finnish. Specifically, we show 1) that zero-shot cross-lingual transfer from the large English CORE corpus can match or surpass previously published monolingual models, and 2) that lightweight monolingual classification requiring very little training data can reach or surpass our zero-shot performance. We further analyse classification results finding that certain registers continue to pose challenges in particular for cross-lingual transfer.