# Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Tasks

arXiv: https://arxiv.org/abs/2406.13469 · HTML: https://mlsanity.com/paper/2406.13469/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 60 references that are in the map.
- First posted on arXiv 19 Jun 2024.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research](https://mlsanity.com/paper/2402.00159/) (2024)
- [ChatGPT: Jack of all trades, master of none](https://mlsanity.com/paper/2302.10724/) (2023)
- [Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT](https://mlsanity.com/paper/2302.10198/) (2023)
- [Uncovering mesa-optimization algorithms in Transformers](https://mlsanity.com/paper/2309.05858/) (2023)
- [Is ChatGPT a Good Sentiment Analyzer? A Preliminary Study](https://mlsanity.com/paper/2304.04339/) (2023)
- [GermanQuAD and GermanDPR: Improving Non-English Question Answering and Passage Retrieval](https://mlsanity.com/paper/2104.12741/) (2021)
- [The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embedding](https://mlsanity.com/paper/2406.02396/) (2024)
- [NorBench -- A Benchmark for Norwegian Language Models](https://mlsanity.com/paper/2305.03880/) (2023)
- [NorNE: Annotating Named Entities for Norwegian](https://mlsanity.com/paper/1911.12146/) (2019)
- [NoReC: The Norwegian Review Corpus](https://mlsanity.com/paper/1710.05370/) (2017)
- [Transfer to a Low-Resource Language via Close Relatives: The Case Study on Faroese](https://mlsanity.com/paper/2304.08823/) (2023)

## Built on by (most cited first, 30 of 34)

- [Theoretical limitations of multi-layer Transformer](https://mlsanity.com/paper/2412.02975/) (2024)
- [Seq vs Seq: An Open Suite of Paired Encoders and Decoders](https://mlsanity.com/paper/2507.11412/) (2025)
- [BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP](https://mlsanity.com/paper/2506.10896/) (2025)
- [Maintaining MTEB: Towards Long Term Usability and Reproducibility of Embedding Benchmarks](https://mlsanity.com/paper/2506.21182/) (2025)
- [MultiWikiQA: A Reading Comprehension Benchmark in 300+ Languages](https://mlsanity.com/paper/2509.04111/) (2025)
- [MTEB-NL and E5-NL: Embedding Benchmark and Models for Dutch](https://mlsanity.com/paper/2509.12340/) (2025)
- [GEITje 7B Ultra: A Conversational Model for Dutch](https://mlsanity.com/paper/2412.04092/) (2024)
- [FoQA: A Faroese Question-Answering Dataset](https://mlsanity.com/paper/2502.07642/) (2025)
- [Improving Child Speech Recognition and Reading Mistake Detection by Using Prompts](https://mlsanity.com/paper/2506.11079/) (2025)
- [Dynaword: From One-shot to Continuously Developed Datasets](https://mlsanity.com/paper/2508.02271/) (2025)
- [Train More Parameters But Mind Their Placement: Insights into Language Adaptation with PEFT](https://mlsanity.com/paper/2412.12674/) (2024)
- [Exploring the Role of Explicit Temporal Modeling in Multimodal Large Language Models for Video Understanding](https://mlsanity.com/paper/2501.16786/) (2025)
- [Standard-to-Dialect Transfer Trends Differ across Text and Speech: A Case Study on Intent and Topic Classification in German Dialects](https://mlsanity.com/paper/2510.07890/) (2025)
- [TabAgent: A Framework for Replacing Agentic Generative Components with Tabular-Textual Classifiers](https://mlsanity.com/paper/2602.16429/) (2026)
- [DeFTX: Denoised Sparse Fine-Tuning for Zero-Shot Cross-Lingual Transfer](https://mlsanity.com/paper/2505.15090/) (2025)
- [Rethinking the Understanding Ability across LLMs through Mutual Information](https://mlsanity.com/paper/2505.23790/) (2025)
- [A Diagnostic Benchmark for Sweden-Related Factual Knowledge](https://mlsanity.com/paper/2510.21360/) (2025)
- [Event Detection with a Context-Aware Encoder and LoRA for Improved Performance on Long-Tailed Classes](https://mlsanity.com/paper/2601.11932/) (2026)
- [EnComp: Lightweight Encoder-Only Context Compression for Retrieval-Augmented Question Answering](https://mlsanity.com/paper/2603.09222/) (2026)
- [Leveraging LLMs for Bangla Grammar Error Correction:Error Categorization, Synthetic Data, and Model Evaluation](https://mlsanity.com/paper/2406.14284/) (2024)
- [Exposing Assumptions in AI Benchmarks through Cognitive Modelling](https://mlsanity.com/paper/2409.16849/) (2024)
- [How to Tune a Multilingual Encoder Model for Germanic Languages: A Study of PEFT, Full Fine-Tuning, and Language Adapters](https://mlsanity.com/paper/2501.06025/) (2025)
- [Automated HIV Screening on Dutch Electronic Health Records with Large Language Models](https://mlsanity.com/paper/2510.19879/) (2025)
- [FIN-bench-v2: A Unified and Robust Benchmark Suite for Evaluating Finnish Large Language Models](https://mlsanity.com/paper/2512.13330/) (2025)
- [Enhancing Multilingual Embeddings via Multi-Way Parallel Text Alignment](https://mlsanity.com/paper/2602.21543/) (2026)
- [Pretraining and Benchmarking Modern Encoders for Latvian](https://mlsanity.com/paper/2603.15005/) (2026)
- [RADAR: Relative Angular Divergence Across Representations](https://mlsanity.com/paper/2605.23028/) (2026)
- [Naturalistic measure of social norms alignment](https://mlsanity.com/paper/2605.23420/) (2026)
- [CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generations](https://mlsanity.com/paper/2605.26293/) (2026)
- [PorTEXTO: A European Portuguese Benchmark for Visual Text Extraction](https://mlsanity.com/paper/2606.19096/) (2026)

## Abstract (arXiv)

This paper explores the performance of encoder and decoder language models on multilingual Natural Language Understanding (NLU) tasks, with a broad focus on Germanic languages. Building upon the ScandEval benchmark, initially restricted to evaluating encoder models, we extend the evaluation framework to include decoder models. We introduce a method for evaluating decoder models on NLU tasks and apply it to the languages Danish, Swedish, Norwegian, Icelandic, Faroese, German, Dutch, and English. Through a series of experiments and analyses, we also address research questions regarding the comparative performance of encoder and decoder models, the impact of NLU task types, and the variation across language resources. Our findings reveal that encoder models can achieve significantly better NLU performance than decoder models despite having orders of magnitude fewer parameters. Additionally, we investigate the correlation between decoders and task performance via a UMAP analysis, shedding light on the unique capabilities of decoder and encoder models. This study contributes to a deeper understanding of language model paradigms in NLU tasks and provides valuable insights for model selection and evaluation in multilingual settings.