mlsanityData to 7 Oct 2026

CUS-QA: Local-Knowledge-Oriented Open-Ended Question Answering Dataset

arXiv 2507.22752 ↗First posted 30 Jul 2025cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

We introduce CUS-QA, a benchmark for evaluation of open-ended regional question answering that encompasses both textual and visual modalities. We also provide strong baselines using state-of-the-art large language models (LLMs). Our dataset consists of manually curated questions and answers grounded in Wikipedia, created by native speakers from Czechia, Slovakia, and Ukraine, with accompanying English translations. It includes both purely textual questions and those requiring visual understanding. We evaluate state-of-the-art LLMs through prompting and add human judgments of answer correctness. Using these human evaluations, we analyze the reliability of existing automatic evaluation metrics. Our baseline results show that even the best open-weight LLMs achieve only over 40% accuracy on textual questions and below 30% on visual questions. LLM-based evaluation metrics show strong correlation with human judgment, while traditional string-overlap metrics perform surprisingly well due to the prevalence of named entities in answers.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 48 of its 75 references that are in the map. First posted on arXiv 30 Jul 2025.

Builds on

48 of 75 references.

References · 48

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2023Mistral 7Bcited by 3,065
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2015VQA: Visual Question Answeringcited by 2,489
  11. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  12. 2025Gemma 3 Technical Reportcited by 1,686
36 more
  1. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  2. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  3. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  4. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  5. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  6. 2024The Faiss librarycited by 370
  7. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 357
  8. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  9. 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 181
  10. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  11. 2023M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Modelscited by 109
  12. 2024EuroLLM: Multilingual Language Models for Europecited by 103
  13. 2024Benchmarking Vision Language Models for Cultural Understandingcited by 100
  14. 2023GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformercited by 84
  15. 2024INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledgecited by 78
  16. 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
  17. 2021Codabench: Flexible, Easy-to-Use and Reproducible Benchmarking Platformcited by 65
  18. 2024All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languagescited by 55
  19. 2024Polos: Multimodal Metric Learning from Human Feedback for Image Captioningcited by 46
  20. 2020MOCHA: A Dataset for Training and Evaluating Generative Reading Comprehension Metricscited by 42
  21. 2023mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMscited by 41
  22. 2024EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Modelscited by 41
  23. 2024WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisinescited by 39
  24. 2024Does Mapo Tofu Contain Coffee? Probing LLMs for Food-related Cultural Knowledgecited by 29
  25. 2025ECLeKTic: a Novel Challenge Set for Evaluation of Cross-Lingual Knowledge Transfercited by 28
  26. 2024BertaQA: How Much Do Language Models Know About Local Culture?cited by 23
  27. 2025MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languagescited by 22
  28. 2024Maya: An Instruction Finetuned Multilingual Multimodal Modelcited by 15
  29. 2024BenCzechMark : A Czech-centric Multitask and Multimetric Benchmark for Large Language Models with Duel Scoring Mechanismcited by 12
  30. 2024DENEB: A Hallucination-Robust Automatic Evaluation Metric for Image Captioningcited by 11
  31. 2024KazQAD: Kazakh Open-Domain Question Answering Datasetcited by 7
  32. 2024Benchmarking Multimodal Models for Ukrainian Language Understanding Across Academic and Cultural Domainscited by 4
  33. 2025What are Foundation Models Cooking in the Post-Soviet World?cited by 3
  34. 2025LAG-MMLU: Benchmarking Frontier LLM Understanding in Latvian and Giriamacited by 2
  35. 2025Towards Multilingual LLM Evaluation for Baltic and Nordic languages: A study on Lithuanian Historycited by 1
  36. 2025Better To Ask in English? Evaluating Factual Accuracy of Multilingual LLMs in English and Low-Resource Languagescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2025Charting the European LLM Benchmarking Landscape: A New Taxonomy and a Set of Best Practicescited by 1
  2. 2026Jako Tako or Fluent? Presenting PoVisLE: A Polish Vision-Language Evaluationcited by 1
  3. 2026LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validationcited by 0
  4. 2026Do LLMs Make More Mistakes If They Do Not Believe the Input Data?cited by 0