mlsanityData to 7 Oct 2026

When LLMs Struggle: Reference-less Translation Evaluation for Low-resource Languages

arXiv 2501.04473 ↗First posted 8 Jan 2025cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

This paper investigates the reference-less evaluation of machine translation for low-resource language pairs, known as quality estimation (QE). Segment-level QE is a challenging cross-lingual language understanding task that provides a quality score (0-100) to the translated output. We comprehensively evaluate large language models (LLMs) in zero/few-shot scenarios and perform instruction fine-tuning using a novel prompt based on annotation guidelines. Our results indicate that prompt-based approaches are outperformed by the encoder-based fine-tuned QE models. Our error analysis reveals tokenization issues, along with errors due to transliteration and named entities, and argues for refinement in LLM pre-training for cross-lingual tasks. We release the data, and models trained publicly for further research.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 23 of its 40 references that are in the map. First posted on arXiv 8 Jan 2025.

Builds on

23 of 40 references.

References · 23

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  7. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  8. 2017Mixed Precision Trainingcited by 1,135
  9. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  10. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  11. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 207
  12. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 163
11 more
  1. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  2. 2023Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Taskcited by 88
  3. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 86
  4. 2020TransQuest: Translation Quality Estimation with Cross-lingual Transformerscited by 59
  5. 2023How Many Demonstrations Do You Need for In-context Learning?cited by 52
  6. 2023Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Promptscited by 45
  7. 2024Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Low-Resource NLPcited by 35
  8. 2024Lost in the Source Language: How Large Language Models Evaluate the Quality of Machine Translationcited by 16
  9. 2023mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequencescited by 10
  10. 2024Towards Large Language Model driven Reference-less Translation Evaluation for English and Indian Languagescited by 3
  11. 2023SurreyAI 2023 Submission for the Quality Estimation Shared Taskcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 3.

7 papers.

  1. 2025Massively Multilingual Adaptation of Large Language Models Using Bilingual Translation Datacited by 4
  2. 2026Human-Centred LLM Privacy Audits: Findings and Frictionscited by 1
  3. 2025FUSE : A Ridge and Random Forest-Based Metric for Evaluating MT in Indigenous Languagescited by 0
  4. 2025LLM-Based Evaluation of Low-Resource Machine Translation: A Reference-less Dialect Guided Approach with a Refined Sylheti-English Benchmarkcited by 0
  5. 2025ALOPE: Adaptive Layer Optimization for Translation Quality Estimation using Large Language Modelscited by 0
  6. 2026Why do Large Language Models Fail in Low-resource Translation? Unraveling the Token Dynamics of Large Language Models for Machine Translationcited by 0
  7. 2026KSE-Web: An Analysis of Hybrid Retrieval and LLM-Assisted Query Expansion for Low-Resource Khmer Semantic Searchcited by 0