mlsanityData to 7 Oct 2026

A Call for Clarity in Reporting BLEU Scores

arXiv 1804.08771 ↗First posted 23 Apr 2018cs.CLcited by 1,790

Not on the map. Why?

Cited by 27 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The field of machine translation faces an under-recognized problem because of inconsistency in the reporting of scores from its dominant metric. Although people refer to "the" BLEU score, BLEU is in fact a parameterized metric whose values can vary wildly with changes to these parameters. These parameters are often not reported or are hard to find, and consequently, BLEU scores between papers cannot be directly compared. I quantify this variation, finding differences as high as 1.8 between commonly used configurations. The main culprit is different tokenization and normalization schemes applied to the reference. Pointing to the success of the parsing community, I suggest machine translation researchers settle upon the BLEU scheme used by the annual Conference on Machine Translation (WMT), which does not allow for user-supplied reference processing, and provide a new tool, SacreBLEU, to facilitate this.

Key facts. Cited by 1,790 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Apr 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 269; 2022: 311; 2023: 388; 2024: 386; 2025: 265; 2026: 170.

1,790 papers.

Sort:

By area:

  1. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  2. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  3. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  4. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  5. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  6. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  7. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  8. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  9. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translationcited by 390
  10. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  11. 2021Random Feature Attentioncited by 350
  12. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
38 more
  1. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  2. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  3. 2023Prompting Large Language Model for Machine Translation: A Case Studycited by 247
  4. 2023Can Large Language Models Transform Computational Social Science?cited by 245
  5. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  6. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  7. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 206
  8. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  9. 2022Continuous diffusion for categorical datacited by 202
  10. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 180
  11. 2022Mega: Moving Average Equipped Gated Attentioncited by 178
  12. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  13. 2021FeTaQA: Free-form Table Question Answeringcited by 174
  14. 2024More Agents Is All You Needcited by 174
  15. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 174
  16. 2022In-context Examples Selection for Machine Translationcited by 172
  17. 2023Is ChatGPT A Good Translator? Yes With GPT-4 As The Enginecited by 172
  18. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 169
  19. 2022Chart-to-Text: A Large-Scale Benchmark for Chart Summarizationcited by 166
  20. 2023Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Promptingcited by 163
  21. 2021To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translationcited by 161
  22. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  23. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 155
  24. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  25. 2023A Reparameterized Discrete Diffusion Model for Text Generationcited by 152
  26. 2021Direct speech-to-speech translation with discrete unitscited by 148
  27. 2022Prompting PaLM for Translation: Assessing Strategies and Performancecited by 144
  28. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  29. 2023Interactive and Explainable Region-guided Radiology Report Generationcited by 139
  30. 2023Towards Making the Most of ChatGPT for Machine Translationcited by 138
  31. 2023Investigating Data Contamination in Modern Benchmarks for Large Language Modelscited by 136
  32. 2023MathDial: A Dialogue Tutoring Dataset with Rich Pedagogical Properties Grounded in Math Reasoning Problemscited by 128
  33. 2023Hallucinations in Large Multilingual Translation Modelscited by 126
  34. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  35. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 123
  36. 2021Textless Speech-to-Speech Translation on Real Datacited by 121
  37. 2022Pretrained Language Models for Text Generation: A Surveycited by 121
  38. 2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112

50 most cited of 1,790 All 1,790 (JSON)