mlsanityData to 7 Oct 2026

Estonian WinoGrande Dataset: Comparative Analysis of LLM Performance on Human and Machine Translation

arXiv 2511.17290 ↗First posted 21 Nov 2025cs.CLcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

In this paper, we present a localized and culturally adapted Estonian translation of the test set from the widely used commonsense reasoning benchmark, WinoGrande. We detail the translation and adaptation process carried out by translation specialists and evaluate the performance of both proprietary and open source models on the human translated benchmark. Additionally, we explore the feasibility of achieving high-quality machine translation by incorporating insights from the manual translation process into the design of a detailed prompt. This prompt is specifically tailored to address both the linguistic characteristics of Estonian and the unique translation challenges posed by the WinoGrande dataset. Our findings show that model performance on the human translated Estonian dataset is slightly lower than on the original English test set, while performance on machine-translated data is notably worse. Additionally, our experiments indicate that prompt engineering offers limited improvement in translation quality or model accuracy, and highlight the importance of involving language specialists in dataset translation and adaptation to ensure reliable and interpretable evaluations of language competency and reasoning in large language models.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 33 of its 52 references that are in the map. First posted on arXiv 21 Nov 2025.

Builds on

33 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Qwen3 Technical Report14 May
  2. 2024Thai Winograd Schemas: A Benchmark for Thai Commonsense Reasoning28 May
  3. 2023Breaking the Language Barrier: Improving Cross-Lingual Reasoning with Structured Self-Attention23 Oct

Other references · 30

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2025Gemma 3 Technical Reportcited by 1,686
  8. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  9. 2018Gender Bias in Coreference Resolutioncited by 438
  10. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  11. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  12. 2024Aya Expanse: Combining Research Breakthroughs for a New Multilingual Frontiercited by 121
18 more
  1. 2024EuroLLM: Multilingual Language Models for Europecited by 103
  2. 2021UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmarkcited by 98
  3. 2020Translation Artifacts in Cross-lingual Transfer Learningcited by 95
  4. 2020RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmarkcited by 53
  5. 2024AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMscited by 46
  6. 2020WinoWhy: A Deep Diagnosis of Essential Commonsense Knowledge for Answering Winograd Schema Challengecited by 43
  7. 2021Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schemacited by 32
  8. 2024mHumanEval -- A Multilingual Benchmark to Evaluate Large Language Models for Code Generationcited by 32
  9. 2019A Surprisingly Robust Trick for Winograd Schema Challengecited by 30
  10. 2024Towards Multilingual LLM Evaluation for European Languagescited by 29
  11. 2022The Defeat of the Winograd Schema Challengecited by 25
  12. 2024Spanish and LLM Benchmarks: is MMLU Lost in Translation?cited by 20
  13. 2022A Warm Start and a Clean Crawled Corpus -- A Recipe for Good Language Modelscited by 13
  14. 2022Slovene SuperGLUE Benchmark: Translation and Evaluationcited by 7
  15. 2025Nile-Chat: Egyptian Language Models for Arabic and Latin Scriptscited by 7
  16. 2024Pipeline Analysis for Developing Instruct LLMs in Low-Resource Languages: A Case Study on Basquecited by 4
  17. 2025D'ej`a Vu: Multilingual LLM Evaluation through the Lens of Machine Translation Evaluationcited by 3
  18. 2024Consolidating and Developing Benchmarking Datasets for the Nepali Natural Language Understanding Taskscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets
  2. 2025The Art of Asking: Multilingual Prompt Optimization for Synthetic Data
  3. 2025MMLU-ProX: A Multilingual Benchmark for Advanced Large Language Model Evaluation
  4. 2025Please Translate Again: Two Simple Experiments on Whether Human-Like Reasoning Helps Translation
  5. 2024What do Large Language Models Need for Machine Translation Evaluation?
  6. 2025Evaluating o1-Like LLMs: Unlocking Reasoning for Translation through Comprehensive Analysis
  7. 2025Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translation
  8. 2025FIN-bench-v2: A Unified and Robust Benchmark Suite for Evaluating Finnish Large Language Models