mlsanityData to 7 Oct 2026

RespondeoQA: a Benchmark for Bilingual Latin-English Question Answering

arXiv 2604.20738 ↗First posted 22 Apr 2026cs.CLcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We introduce a benchmark dataset for question answering and translation in bilingual Latin and English settings, containing about 7,800 question-answer pairs. The questions are drawn from Latin pedagogical sources, including exams, quizbowl-style trivia, and textbooks ranging from the 1800s to the present. After automated extraction, cleaning, and manual review, the dataset covers a diverse range of question types: knowledge- and skill-based, multihop reasoning, constrained translation, and mixed language pairs. To our knowledge, this is the first QA benchmark centered on Latin. As a case study, we evaluate three large language models -- LLaMa 3, Qwen QwQ, and OpenAI's o3-mini -- finding that all perform worse on skill-oriented questions. Although the reasoning models perform better on scansion and literary-device tasks, they offer limited improvement overall. QwQ performs slightly better on questions asked in Latin, but LLaMa3 and o3-mini are more task dependent. This dataset provides a new resource for assessing model capabilities in a specialized linguistic and cultural domain, and the creation process can be easily adapted for other languages. The dataset is available at: https://github.com/slanglab/RespondeoQA

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 22 of its 53 references that are in the map. First posted on arXiv 22 Apr 2026.

Builds on

22 of 53 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation13 Mar

Other references · 21

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  4. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  5. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  6. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  7. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
  8. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 300
  9. 2020Latin BERT: A Contextual Language Model for Classical Philologycited by 43
  10. 2019Quizbowl: The Case for Incremental Question Answeringcited by 36
  11. 2022DEMETR: Diagnosing Evaluation Metrics for Translationcited by 36
  12. 2020Do Multi-Hop Question Answering Systems Know How to Answer the Single-Hop Sub-Questions?cited by 29
9 more
  1. 2024Towards Multilingual LLM Evaluation for European Languagescited by 29
  2. 2025Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Trainingcited by 19
  3. 2021BiRdQA: A Bilingual Dataset for Question Answering on Tricky Riddlescited by 10
  4. 2023Exploring Large Language Models for Classical Philologycited by 10
  5. 2024Sui Generis: Large Language Models for Authorship Attribution and Verification in Latincited by 8
  6. 2025Institutional Books 1.0: A 242B token dataset from Harvard Library's collections, refined for accuracy and usabilitycited by 7
  7. 2025Explicit Learning and the LLM in Machine Translationcited by 6
  8. 2022Pir'a: A Bilingual Portuguese-English Dataset for Question-Answering about the Oceancited by 4
  9. 2025PersianMedQA: Evaluating Large Language Models on a Persian-English Bilingual Medical Question Answering Benchmarkcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023On the Calibration of Multilingual Question Answering LLMs
  2. 2024NativQA: Multilingual Culturally-Aligned Natural Query for LLMs
  3. 2023Bridging the Language Gap: Knowledge Injected Multilingual Question Answering
  4. 2024SynDARin: Synthesising Datasets for Automated Reasoning in Low-Resource Languages
  5. 2024The Power of Question Translation Training in Multilingual Reasoning: Broadened Scope and Deepened Insights
  6. 2025Multilingual Question Answering in Low-Resource Settings: A Dzongkha-English Benchmark for Foundation Models
  7. 2025JGU Mainz's Submission to the WMT25 Shared Task on LLMs with Limited Resources for Slavic Languages: MT and QA
  8. 2024Question Translation Training for Better Multilingual Reasoning