# Measuring Harmful Representations in Scandinavian Language Models

arXiv: https://arxiv.org/abs/2211.11678 · HTML: https://mlsanity.com/paper/2211.11678/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 42 references that are in the map.
- First posted on arXiv 21 Nov 2022.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [Multimodal datasets: misogyny, pornography, and malignant stereotypes](https://mlsanity.com/paper/2110.01963/) (2021)
- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [Social Biases in NLP Models as Barriers for Persons with Disabilities](https://mlsanity.com/paper/2005.00813/) (2020)
- [Assessing Gender Bias in Machine Translation -- A Case Study with Google Translate](https://mlsanity.com/paper/1809.02208/) (2018)
- [Evaluating the Underlying Gender Bias in Contextualized Word Embeddings](https://mlsanity.com/paper/1904.08783/) (2019)
- [A Survey on Gender Bias in Natural Language Processing](https://mlsanity.com/paper/2112.14168/) (2021)
- [A Survey of Race, Racism, and Anti-Racism in NLP](https://mlsanity.com/paper/2106.11410/) (2021)
- [On Measuring Gender Bias in Translation of Gender-neutral Pronouns](https://mlsanity.com/paper/1905.11684/) (2019)
- [Offensive Language and Hate Speech Detection for Danish](https://mlsanity.com/paper/1908.04531/) (2019)
- [What the [MASK]? Making Sense of Language-Specific BERT Models](https://mlsanity.com/paper/2003.02912/) (2020)
- [Investigating Gender Bias in BERT](https://mlsanity.com/paper/2009.05021/) (2020)
- [Operationalizing a National Digital Library: The Case for a Norwegian Transformer Model](https://mlsanity.com/paper/2104.09617/) (2021)
- [Large-Scale Contextualised Language Modelling for Norwegian](https://mlsanity.com/paper/2104.06546/) (2021)
- [Good Secretaries, Bad Truck Drivers? Occupational Gender Stereotypes in Sentiment Analysis](https://mlsanity.com/paper/1906.10256/) (2019)
- [Addressing Exposure Bias With Document Minimum Risk Training: Cambridge at the WMT20 Biomedical Translation Task](https://mlsanity.com/paper/2010.05333/) (2020)
- [The State of Profanity Obfuscation in Natural Language Processing](https://mlsanity.com/paper/2210.07595/) (2022)

## Built on by (most cited first, 7 of 7)

- [Social Bias Probing: Fairness Benchmarking for Language Models](https://mlsanity.com/paper/2311.09090/) (2023)
- [NorBench -- A Benchmark for Norwegian Language Models](https://mlsanity.com/paper/2305.03880/) (2023)
- [Measuring Gender Bias in West Slavic Language Models](https://mlsanity.com/paper/2304.05783/) (2023)
- [Beyond Traditional Teaching: The Potential of Large Language Models and Chatbots in Graduate Engineering Education](https://mlsanity.com/paper/2309.13059/) (2023)
- [Measuring Normative and Descriptive Biases in Language Models Using Census Data](https://mlsanity.com/paper/2304.05764/) (2023)
- [Social Bias in Multilingual Language Models: A Survey](https://mlsanity.com/paper/2508.20201/) (2025)
- [Learning Horn Envelopes via Queries from Large Language Models](https://mlsanity.com/paper/2305.12143/) (2023)

## Abstract (arXiv)

Scandinavian countries are perceived as role-models when it comes to gender equality. With the advent of pre-trained language models and their widespread usage, we investigate to what extent gender-based harmful and toxic content exist in selected Scandinavian language models. We examine nine models, covering Danish, Swedish, and Norwegian, by manually creating template-based sentences and probing the models for completion. We evaluate the completions using two methods for measuring harmful and toxic completions and provide a thorough analysis of the results. We show that Scandinavian pre-trained language models contain harmful and gender-based stereotypes with similar values across all languages. This finding goes against the general expectations related to gender equality in Scandinavian countries and shows the possible problematic outcomes of using such models in real-world settings.