# Evaluating Monolingual and Multilingual Large Language Models for Greek Question Answering: The DemosQA Benchmark

arXiv: https://arxiv.org/abs/2602.16811 · HTML: https://mlsanity.com/paper/2602.16811/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 45 references that are in the map.
- First posted on arXiv 18 Feb 2026.

## Builds on (most cited first, 27 of 27)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [A Comprehensive Overview of Large Language Models](https://mlsanity.com/paper/2307.06435/) (2023)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [How is ChatGPT's behavior changing over time?](https://mlsanity.com/paper/2307.09009/) (2023)
- [The case for 4-bit precision: k-bit Inference Scaling Laws](https://mlsanity.com/paper/2212.09720/) (2022)
- [The Effect of Sampling Temperature on Problem Solving in Large Language Models](https://mlsanity.com/paper/2402.05201/) (2024)
- [Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontier](https://mlsanity.com/paper/2412.04261/) (2024)
- [Systematic Inequalities in Language Technology Performance across the World's Languages](https://mlsanity.com/paper/2110.06733/) (2021)
- [INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge](https://mlsanity.com/paper/2411.19799/) (2024)
- [Command A: An Enterprise-Ready Large Language Model](https://mlsanity.com/paper/2504.00698/) (2025)
- [Translationese in Machine Translation Evaluation](https://mlsanity.com/paper/1906.09833/) (2019)
- [A Study on Large Language Models' Limitations in Multiple-Choice Question Answering](https://mlsanity.com/paper/2401.07955/) (2024)
- [Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)](https://mlsanity.com/paper/2505.21091/) (2025)
- [Plutus: Benchmarking Large Language Models in Low-Resource Greek Finance](https://mlsanity.com/paper/2502.18772/) (2025)
- [Meltemi: The first open Large Language Model for Greek](https://mlsanity.com/paper/2407.20743/) (2024)
- [GreekBarBench: A Challenging Benchmark for Free-Text Legal Reasoning and Citations](https://mlsanity.com/paper/2505.17267/) (2025)
- [Krikri: Advancing Open Large Language Models for Greek](https://mlsanity.com/paper/2505.13772/) (2025)
- [The anatomy of Reddit: An overview of academic research](https://mlsanity.com/paper/1810.10881/) (2018)
- [NLP for The Greek Language: A Longer Survey](https://mlsanity.com/paper/2408.10962/) (2024)
- [A Systematic Survey of Natural Language Processing for the Greek Language](https://mlsanity.com/paper/2407.09861/) (2024)
- [Open or Closed LLM for Lesser-Resourced Languages? Lessons from Greek](https://mlsanity.com/paper/2501.12826/) (2025)

## Built on by (most cited first, 2 of 2)

- [Maistros: A Greek Large Language Model Adapted Through Knowledge Distillation From Large Reasoning Models](https://mlsanity.com/paper/2605.01870/) (2026)
- [Pass or Fail? Evaluating LLMs on Two Greek Examination Benchmarks](https://mlsanity.com/paper/2609.34800/) (2026)

## Abstract (arXiv)

Recent advancements in Natural Language Processing and Deep Learning have enabled the development of Large Language Models (LLMs), which have significantly advanced the state-of-the-art across a wide range of tasks, including Question Answering (QA). Despite these advancements, research on LLMs has primarily targeted high-resourced languages (e.g., English), and only recently has attention shifted toward multilingual models. However, these models demonstrate a training data bias towards a small number of popular languages or rely on transfer learning from high- to under-resourced languages; this may lead to a misrepresentation of social, cultural, and historical aspects. To address this challenge, monolingual LLMs have been developed for under-resourced languages; however, their effectiveness remains less studied when compared to multilingual counterparts on language-specific tasks. In this study, we address this research gap in Greek QA by contributing: (i) DemosQA, a novel dataset, which is constructed using social media user questions and community-reviewed answers to better capture the Greek social and cultural zeitgeist; (ii) a memory-efficient LLM evaluation framework adaptable to diverse QA datasets and languages; and (iii) an extensive evaluation of 11 monolingual and multilingual LLMs on 6 human-curated Greek QA datasets using 3 different prompting strategies. We release our code and data to facilitate reproducibility.