mlsanityData to 14 Oct 2026

Evaluating OpenAI GPT Models for Translation of Endangered Uralic Languages: A Comparison of Reasoning and Non-Reasoning Architectures

arXiv 2512.16287 ↗First posted 18 Dec 2025cs.CLcited by 2

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

The evaluation of Large Language Models (LLMs) for translation tasks has primarily focused on high-resource languages, leaving a significant gap in understanding their performance on low-resource and endangered languages. This study presents a comprehensive comparison of OpenAI's GPT models, specifically examining the differences between reasoning and non-reasoning architectures for translating between Finnish and four low-resource Uralic languages: Komi-Zyrian, Moksha, Erzya, and Udmurt. Using a parallel corpus of literary texts, we evaluate model willingness to attempt translation through refusal rate analysis across different model architectures. Our findings reveal significant performance variations between reasoning and non-reasoning models, with reasoning models showing 16 percentage points lower refusal rates. The results provide valuable insights for researchers and practitioners working with Uralic languages and contribute to the broader understanding of reasoning model capabilities for endangered language preservation.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 11 of its 28 references that are in the map. First posted on arXiv 18 Dec 2025.

Builds on

11 of 28 references.

References · 11

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  5. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 469
  6. 2022The first neural machine translation system for the Erzya languagecited by 7
  7. 2021Neural Morphology Dataset and Models for Multiple Languages, from the Large to the Endangeredcited by 5
  8. 2021The Current State of Finnish NLPcited by 5
  9. 2025A Comparative Analysis of Ethical and Safety Gaps in LLMs using Relative Danger Coefficientcited by 2
  10. 2023Sentiment Analysis Using Aligned Word Embeddings for Uralic Languagescited by 1
  11. 2025Efficient Toxicity Detection in Gaming Chats: A Comparative Study of Embeddings, Fine-Tuned Transformers and LLMscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Tokenization and Morphological Fidelity in Uralic NLP: A Cross-Lingual Evaluationcited by 0
  2. 2026A Komi-Yazva--Russian Parallel Corpus and Evaluation Protocol for Zero- and Few-Shot LLM Translationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023GPT-Fathom: Benchmarking Large Language Models to Decipher the Evolutionary Path towards GPT-4 and Beyond
  2. 2025Is GPT-OSS Good? A Comprehensive Evaluation of OpenAI's Latest Open Source Models
  3. 2025Evaluating o1-Like LLMs: Unlocking Reasoning for Translation through Comprehensive Analysis
  4. 2026Scaling Multilingual Safety and Reasoning Evaluation through Reasoning-Guided Translation
  5. 2024SUTRA: Scalable Multilingual Language Model Architecture
  6. 2025Estonian WinoGrande Dataset: Comparative Analysis of LLM Performance on Human and Machine Translation
  7. 2025LLaMAX2: Your Translation-Enhanced Model also Performs Well in Reasoning
  8. 2025Why Do Multilingual Reasoning Gaps Emerge in Reasoning Language Models?