mlsanityData to 7 Oct 2026

Dual Conditional Cross-Entropy Filtering of Noisy Parallel Corpora

arXiv 1809.00197 ↗First posted 1 Sep 2018cs.CLcited by 40

Not on the map. Why?

Abstract (arXiv)

In this work we introduce dual conditional cross-entropy filtering for noisy parallel data. For each sentence pair of the noisy parallel corpus we compute cross-entropy scores according to two inverse translation models trained on clean data. We penalize divergent cross-entropies and weigh the penalty by the cross-entropy average of both models. Sorting or thresholding according to these scores results in better subsets of parallel data. We achieve higher BLEU scores with models trained on parallel data filtered only from Paracrawl than with models trained on clean WMT data. We further evaluate our method in the context of the WMT2018 shared task on parallel corpus filtering and achieve the overall highest ranking scores of the shared task, scoring top in three out of four subtasks.

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Sep 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 13; 2023: 7; 2024: 5; 2025: 1; 2026: 1.

40 papers.

Sort:

By area:

  1. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  2. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 241
  3. 2021The Curious Case of Hallucinations in Neural Machine Translationcited by 161
  4. 2021Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languagescited by 133
  5. 2021Survey of Low-Resource Machine Translationcited by 92
  6. 2021Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translationcited by 56
  7. 2022Data Scaling Laws in NMT: The Effect of Noise and Architecturecited by 55
  8. 2021Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Surveycited by 46
  9. 2021Prevent the Language Model from being Overconfident in Neural Machine Translationcited by 34
  10. 2022Speaking Multiple Languages Affects the Moral Bias of Language Modelscited by 34
  11. 2024A Shocking Amount of the Web is Machine Translated: Insights from Multi-Way Parallelismcited by 33
  12. 2022Faithfulness in Natural Language Generation: A Systematic Survey of Analysis, Evaluation and Optimization Methodscited by 27
28 more
  1. 2023There's no Data Like Better Data: Using QE Metrics for MT Data Filteringcited by 19
  2. 2023Make Every Example Count: On the Stability and Utility of Self-Influence for Learning from Noisy NLP Datasetscited by 18
  3. 2024Task-Adaptive Pretrained Language Models via Clustered-Importance Samplingcited by 18
  4. 2023Domain Terminology Integration into Machine Translation: Leveraging Large Language Modelscited by 17
  5. 2022Data Selection Curriculum for Neural Machine Translationcited by 11
  6. 2022Non-Autoregressive Machine Translation: It's Not as Fast as it Seemscited by 11
  7. 2024Critical Learning Periods: Leveraging Early Training Dynamics for Efficient Data Pruningcited by 10
  8. 2025Reference-free Evaluation Metrics for Text Generation: A Surveycited by 9
  9. 2021Gradient-guided Loss Masking for Neural Machine Translationcited by 8
  10. 2023Learning Multilingual Sentence Representations with Cross-lingual Consistency Regularizationcited by 7
  11. 2022NMTScore: A Multilingual Analysis of Translation-based Text Similarity Measurescited by 6
  12. 2022Separating Grains from the Chaff: Using Data Filtering to Improve Multilingual Translation for Low-Resourced African Languagescited by 6
  13. 2021Cross-language Sentence Selection via Data Augmentation and Rationale Trainingcited by 5
  14. 2021Lesan -- Machine Translation for Low Resource Languagescited by 5
  15. 2022Bitext Mining for Low-Resource Languages via Contrastive Learningcited by 5
  16. 2021HintedBT: Augmenting Back-Translation with Quality and Transliteration Hintscited by 3
  17. 2022Analyzing the Use of Influence Functions for Instance-Specific Data Filtering in Neural Machine Translationcited by 3
  18. 2024Machine Translation Models are Zero-Shot Detectors of Translation Directioncited by 3
  19. 2022CUNI Systems for the WMT22 Czech-Ukrainian Translation Taskcited by 2
  20. 2023Robustness of Multi-Source MT to Transcription Errorscited by 2
  21. 2024Charles Translator: A Machine Translation System between Ukrainian and Czechcited by 2
  22. 2021Learning Feature Weights using Reward Modeling for Denoising Parallel Corporacited by 1
  23. 2021Generalised Unsupervised Domain Adaptation of Neural Machine Translation with Cross-Lingual Data Selectioncited by 1
  24. 2022Can Synthetic Translations Improve Bitext Quality?cited by 1
  25. 2022High-Resource Methodological Bias in Low-Resource Investigationscited by 1
  26. 2023Separating the Wheat from the Chaff with BREAD: An open-source benchmark and metrics to detect redundancy in textcited by 1
  27. 2026MERIT: Multilingual Expert-Reward Informed Tuning for Chinese-Centric Low-Resource Machine Translationcited by 1
  28. 2023SignBank+: Preparing a Multilingual Sign Language Dataset for Machine Translation Using Large Language Modelscited by 0