mlsanityData to 14 Oct 2026

BERTTune: Fine-Tuning Neural Machine Translation with BERTScore

arXiv 2106.02208 ↗First posted 4 Jun 2021cs.CLcited by 18

Not on the map. Why?

Abstract (arXiv)

Neural machine translation models are often biased toward the limited translation references seen during training. To amend this form of overfitting, in this paper we propose fine-tuning the models with a novel training objective based on the recently-proposed BERTScore evaluation metric. BERTScore is a scoring function based on contextual embeddings that overcomes the typical limitations of n-gram-based metrics (e.g. synonyms, paraphrases), allowing translations that are different from the references, yet close in the contextual embedding space, to be treated as substantially correct. To be able to use BERTScore as a training objective, we propose three approaches for generating soft predictions, allowing the network to remain completely differentiable end-to-end. Experiments carried out over four, diverse language pairs have achieved improvements of up to 0.58 pp (3.28%) in BLEU score and up to 0.76 pp (0.98%) in BERTScore (F_BERT) when fine-tuning a strong baseline.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 27 of its 36 references that are in the map. First posted on arXiv 4 Jun 2021.

Builds on

27 of 36 references.

References · 27

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  7. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  8. 2018Deep contextualized word representationscited by 2,290
  9. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  10. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
  11. 2015Sequence Level Training with Recurrent Neural Networkscited by 576
  12. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 418
15 more
  1. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 341
  2. 2017Six Challenges for Neural Machine Translationcited by 335
  3. 2016Sequence-to-Sequence Learning as Beam-Search Optimizationcited by 127
  4. 2018Document-Level Neural Machine Translation with Hierarchical Attention Networkscited by 85
  5. 2017Classical Structured Prediction Losses for Sequence to Sequence Learningcited by 55
  6. 2019Towards Making the Most of BERT in Neural Machine Translationcited by 43
  7. 2020Language Model Prior for Low-Resource Neural Machine Translationcited by 37
  8. 2019Pre-trained Language Model Representations for Language Generationcited by 34
  9. 2018Von Mises-Fisher Loss for Training Sequence to Sequence Models with Continuous Outputscited by 27
  10. 2019Deep Reinforcement Learning with Distributional Semantic Rewards for Abstractive Summarizationcited by 24
  11. 2019On the use of BERT for Neural Machine Translationcited by 23
  12. 2018Bag-of-Words as Target for Neural Machine Translationcited by 11
  13. 2018Token-level and sequence-level loss smoothing for RNN language modelscited by 6
  14. 2018Training Neural Machine Translation using Word Embedding-based Losscited by 2
  15. 2019ReWE: Regressing Word Embeddings for Regularization of Neural Machine Translation Systemscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 7; 2023: 5; 2024: 1; 2025: 2; 2026: 2.

18 papers.

Sort:

By area:

  1. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 238
  2. 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 138
  3. 2022Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decodingcited by 54
  4. 2022EditEval: An Instruction-Based Benchmark for Text Improvementscited by 30
  5. 2022Rainproof: An Umbrella To Shield Text Generators From Out-Of-Distribution Datacited by 21
  6. 2023Towards Explainable Evaluation Metrics for Machine Translationcited by 15
  7. 2022SESCORE2: Learning Text Generation Evaluation via Synthesizing Realistic Mistakescited by 9
  8. 2023T3L: Translate-and-Test Transfer Learning for Cross-Lingual Text Classificationcited by 8
  9. 2023N-Critics: Self-Refinement of Large Language Models with Ensemble of Criticscited by 8
  10. 2025SGIC: A Self-Guided Iterative Calibration Framework for RAGcited by 8
  11. 2023Translating away Translationese without Parallel Datacited by 5
  12. 2022Few-shot Question Generation for Personalized Feedback in Intelligent Tutoring Systemscited by 4
6 more
  1. 2025SignAligner: Harmonizing Complementary Pose Modalities for Coherent Sign Language Generationcited by 3
  2. 2022Universal Evasion Attacks on Summarization Scoringcited by 2
  3. 2024An Empirical Study on Self-correcting Large Language Models for Data Science Code Generationcited by 2
  4. 2022Soft Alignment Objectives for Robust Adaptation of Language Generationcited by 0
  5. 2026DTO: a Differentiable Training Objective for Effective Counterfactual Story Rewritingcited by 0
  6. 2026Data Efficient Sample Selection for In-Context Learningcited by 0