mlsanityData to 14 Oct 2026

Does BLEU Score Work for Code Migration?

arXiv 1906.04903 ↗First posted 12 Jun 2019cs.SE cs.CLcited by 49

Not on the map. Why?

Abstract (arXiv)

Statistical machine translation (SMT) is a fast-growing sub-field of computational linguistics. Until now, the most popular automatic metric to measure the quality of SMT is BiLingual Evaluation Understudy (BLEU) score. Lately, SMT along with the BLEU metric has been applied to a Software Engineering task named code migration. (In)Validating the use of BLEU score could advance the research and development of SMT-based code migration tools. Unfortunately, there is no study to approve or disapprove the use of BLEU score for source code. In this paper, we conducted an empirical study on BLEU score to (in)validate its suitability for the code migration task due to its inability to reflect the semantics of source code. In our work, we use human judgment as the ground truth to measure the semantic correctness of the migrated code. Our empirical study demonstrates that BLEU does not reflect translation quality due to its weak correlation with the semantic correctness of translated code. We provided counter-examples to show that BLEU is ineffective in comparing the translation quality between SMT-based models. Due to BLEU's ineffectiveness for code migration task, we propose an alternative metric RUBY, which considers lexical, syntactical, and semantic representations of source code. We verified that RUBY achieves a higher correlation coefficient with the semantic correctness of migrated code, 0.775 in comparison with 0.583 of BLEU score. We also confirmed the effectiveness of RUBY in reflecting the changes in translation quality of SMT-based translation models. With its advantages, RUBY can be used to evaluate SMT-based code migration models.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). First posted on arXiv 12 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 6; 2023: 9; 2024: 12; 2025: 15; 2026: 4.

49 papers.

Sort:

By area:

  1. 2022GitHub Copilot AI pair programmer: Asset or Liability?cited by 152
  2. 2025Are VLMs Ready for Autonomous Driving? An Empirical Study from the Reliability, Data, and Metric Perspectivescited by 122
  3. 2022Out of the BLEU: how should we assess quality of the Code Generation models?cited by 65
  4. 2023CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translationcited by 57
  5. 2023ICE-Score: Instructing Large Language Models to Evaluate Codecited by 52
  6. 2024CodeJudge: Evaluating Code Generation with Large Language Modelscited by 44
  7. 2023Data Contamination Through the Lens of Timecited by 40
  8. 2024Code Comprehension then Auditing for Unsupervised LLM Evaluationcited by 32
  9. 2024Consistent Autoformalization for Constructing Mathematical Librariescited by 21
  10. 2024Benchmarks and Metrics for Evaluations of Code Generation: A Critical Reviewcited by 20
  11. 2024NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Promptscited by 19
  12. 2025Don't Judge Code by Its Cover: Exploring Biases in LLM Judges for Code Evaluationcited by 18
37 more
  1. 2021Shellcode_IA32: A Dataset for Automatic Shellcode Generationcited by 16
  2. 2024Improving Retrieval-augmented Text-to-SQL with AST-based Ranking and Schema Pruningcited by 16
  3. 2022Beyond the C: Retargetable Decompilation using Neural Machine Translationcited by 15
  4. 2022Who Evaluates the Evaluators? On Automatic Metrics for Assessing AI-based Offensive Code Generatorscited by 11
  5. 2024RAMBO: Enhancing RAG-based Repository-Level Method Body Completioncited by 10
  6. 2021Toward Code Generation: A Survey and Lessons from Semantic Parsingcited by 8
  7. 2024CRScore: Grounding Automated Evaluation of Code Review Comments in Code Claims and Smellscited by 8
  8. 2021NeurIPS 2020 NLC2CMD Competition: Translating Natural Language to Bash Commandscited by 7
  9. 2023Assessing Evaluation Metrics for Neural Test Oracle Generationcited by 7
  10. 2023Automating the Correctness Assessment of AI-generated Code for Security Contextscited by 6
  11. 2024On the Limitations of Embedding Based Methods for Measuring Functional Correctness for Code Generationcited by 6
  12. 2025Beyond Gold Standards: Epistemic Ensemble of LLM Judges for Formal Mathematical Reasoningcited by 6
  13. 2025MASA: LLM-Driven Multi-Agent Systems for Autoformalizationcited by 6
  14. 2024Enhancing Source Code Security with LLMs: Demystifying The Challenges and Generating Reliable Repairscited by 4
  15. 2023NL2CMD: An Updated Workflow for Natural Language to Bash Commands Translationcited by 3
  16. 2023Neural Machine Translation for Code Generationcited by 3
  17. 2025Deep-Bench: Deep Learning Benchmark Dataset for Code Generationcited by 3
  18. 2022Can NMT Understand Me? Towards Perturbation-based Evaluation of NMT Models for Code Generationcited by 2
  19. 2023InstructPipe: Generating Visual Blocks Pipelines with Human Instructions and LLMscited by 2
  20. 2024NoviCode: Generating Programs from Natural Language Utterances by Novicescited by 2
  21. 2025Fast, Fine-Grained Equivalence Checking for Neural Decompilerscited by 2
  22. 2025Measuring LLM Code Generation Stability via Structural Entropycited by 2
  23. 2025David vs. Goliath: A comparative study of different-sized LLMs for code generation in the domain of automotive scenario generationcited by 1
  24. 2025MATCH: Task-Driven Code Evaluation through Contrastive Learningcited by 1
  25. 2025Physics-Based Benchmarking Metrics for Multimodal Synthetic Imagescited by 1
  26. 2025Modeling Romanized Hindi and Bengali: Dataset Creation and Multilingual LLM Integrationcited by 1
  27. 2025Benchmarking Code LLM-as-a-Judge via Constrained Perturbation and Calibrationcited by 1
  28. 2026Beyond BLEU: A Semantic Evaluation Method for Code Translationcited by 1
  29. 2022Can We Generate Shellcodes via Natural Language? An Empirical Studycited by 0
  30. 2023Leveraging Deep Learning for Abstractive Code Summarization of Unofficial Documentationcited by 0
  31. 2024Can Neural Decompilation Assist Vulnerability Prediction on Binary Code?cited by 0
  32. 2025Learning Bug Context for PyTorch-to-JAX Translation with LLMscited by 0
  33. 2025Excision Score: Evaluating Edits with Surgical Precisioncited by 0
  34. 2025Asm2SrcEval: Evaluating Large Language Models for Assembly-to-Source Code Translationcited by 0
  35. 2026Leveraging Generative AI for Enhancing Domain-Driven Software Designcited by 0
  36. 2026Can Code Evaluation Metrics Detect Code Plagiarism?cited by 0
  37. 2026Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse Engineeringcited by 0