# RespondeoQA: a Benchmark for Bilingual Latin-English Question Answering

arXiv: https://arxiv.org/abs/2604.20738 · HTML: https://mlsanity.com/paper/2604.20738/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 22 of its 53 references that are in the map.
- First posted on arXiv 22 Apr 2026.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 22 of 22)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages](https://mlsanity.com/paper/2003.05002/) (2020)
- [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://mlsanity.com/paper/1910.07475/) (2019)
- [The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants](https://mlsanity.com/paper/2308.16884/) (2023)
- [MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation](https://mlsanity.com/paper/2503.10497/) (2025)
- [Latin BERT: A Contextual Language Model for Classical Philology](https://mlsanity.com/paper/2009.10053/) (2020)
- [Quizbowl: The Case for Incremental Question Answering](https://mlsanity.com/paper/1904.04792/) (2019)
- [DEMETR: Diagnosing Evaluation Metrics for Translation](https://mlsanity.com/paper/2210.13746/) (2022)
- [Do Multi-Hop Question Answering Systems Know How to Answer the Single-Hop Sub-Questions?](https://mlsanity.com/paper/2002.09919/) (2020)
- [Towards Multilingual LLM Evaluation for European Languages](https://mlsanity.com/paper/2410.08928/) (2024)
- [Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training](https://mlsanity.com/paper/2506.01732/) (2025)
- [BiRdQA: A Bilingual Dataset for Question Answering on Tricky Riddles](https://mlsanity.com/paper/2109.11087/) (2021)
- [Exploring Large Language Models for Classical Philology](https://mlsanity.com/paper/2305.13698/) (2023)
- [Sui Generis: Large Language Models for Authorship Attribution and Verification in Latin](https://mlsanity.com/paper/2410.09245/) (2024)
- [Institutional Books 1.0: A 242B token dataset from Harvard Library's collections, refined for accuracy and usability](https://mlsanity.com/paper/2506.08300/) (2025)
- [Explicit Learning and the LLM in Machine Translation](https://mlsanity.com/paper/2503.09454/) (2025)
- [Pir'a: A Bilingual Portuguese-English Dataset for Question-Answering about the Ocean](https://mlsanity.com/paper/2202.02398/) (2022)
- [PersianMedQA: Evaluating Large Language Models on a Persian-English Bilingual Medical Question Answering Benchmark](https://mlsanity.com/paper/2506.00250/) (2025)

## Abstract (arXiv)

We introduce a benchmark dataset for question answering and translation in bilingual Latin and English settings, containing about 7,800 question-answer pairs. The questions are drawn from Latin pedagogical sources, including exams, quizbowl-style trivia, and textbooks ranging from the 1800s to the present. After automated extraction, cleaning, and manual review, the dataset covers a diverse range of question types: knowledge- and skill-based, multihop reasoning, constrained translation, and mixed language pairs. To our knowledge, this is the first QA benchmark centered on Latin. As a case study, we evaluate three large language models -- LLaMa 3, Qwen QwQ, and OpenAI's o3-mini -- finding that all perform worse on skill-oriented questions. Although the reasoning models perform better on scansion and literary-device tasks, they offer limited improvement overall. QwQ performs slightly better on questions asked in Latin, but LLaMa3 and o3-mini are more task dependent. This dataset provides a new resource for assessing model capabilities in a specialized linguistic and cultural domain, and the creation process can be easily adapted for other languages. The dataset is available at: https://github.com/slanglab/RespondeoQA