mlsanityData to 13 Oct 2026

When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLP

arXiv 2303.16166 ↗First posted 28 Mar 2023cs.CL cs.AIcited by 14

Not on the map. Why?

Abstract (arXiv)

Despite its crucial role in research experiments, code correctness is often presumed only on the basis of the perceived quality of results. This assumption comes with the risk of erroneous outcomes and potentially misleading findings. To address this issue, we posit that the current focus on reproducibility should go hand in hand with the emphasis on software quality. We present a case study in which we identify and fix three bugs in widely used implementations of the state-of-the-art Conformer architecture. Through experiments on speech recognition and translation in various languages, we demonstrate that the presence of bugs does not prevent the achievement of good and reproducible results, which however can lead to incorrect conclusions that potentially misguide future research. As a countermeasure, we propose a Code-quality Checklist and release pangoliNN, a library dedicated to testing neural models, with the goal of promoting coding best practices and improving research software quality within the NLP community.

Key facts. Cited by 14 papers in the map (data through 13 Oct 2026). Builds on 48 of its 115 references that are in the map. First posted on arXiv 28 Mar 2023.

Builds on

48 of 115 references.

References · 48

  1. 2017Attention Is All You Needcited by 60,106
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,093
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,223
  4. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,991
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,910
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,979
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,844
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,835
  9. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,793
  10. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,359
  11. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,248
  12. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 992
36 more
  1. 2016Language Modeling with Gated Convolutional Networkscited by 694
  2. 2019Green AIcited by 623
  3. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
  4. 2020Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)cited by 319
  5. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 232
  6. 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 175
  7. 2019Show Your Work: Improved Reporting of Experimental Resultscited by 166
  8. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  9. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 147
  10. 2020Recent Developments on ESPnet Toolkit Boosted by Conformercited by 132
  11. 2017Sequence-to-Sequence Models Can Directly Translate Foreign Speechcited by 121
  12. 2016Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translationcited by 113
  13. 2019Understanding and Improving Transformer From a Multi-Particle Dynamic System Point of Viewcited by 109
  14. 2021Do Transformer Modifications Transfer Across Implementations and Applications?cited by 87
  15. 2021TorchAudio: Building Blocks for Audio and Speech Processingcited by 83
  16. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
  17. 2019A Step Toward Quantifying Independently Reproducible Machine Learning Researchcited by 77
  18. 2021Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Paperscited by 71
  19. 2014End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Resultscited by 68
  20. 2020Bridging the Modality Gap for Speech-to-Text Translationcited by 55
  21. 2021Stacked Acoustic-and-Textual Encoding: Integrating the Pre-trained Models into Speech Translation Encoderscited by 48
  22. 2021Just What do You Think You're Doing, Dave?' A Checklist for Responsible Data Use in NLPcited by 47
  23. 2021CTC-based Compression for Direct Speech Translationcited by 45
  24. 2021A Systematic Review of Reproducibility Research in Natural Language Processingcited by 34
  25. 2022Revisiting End-to-End Speech-to-Text Translation From Scratchcited by 26
  26. 2021A large-scale study on research code quality and executioncited by 22
  27. 2021Conformer-Based Self-Supervised Learning for Non-Speech Audio Taskscited by 22
  28. 2021Speechformer: Reducing Information Loss in Direct Speech Translationcited by 15
  29. 2022Efficient yet Competitive Speech Translation: FBK@IWSLT2022cited by 14
  30. 2017Stronger Baselines for Trustable Results in Neural Machine Translationcited by 12
  31. 2020On Target Segmentation for Direct Speech Translationcited by 12
  32. 2021Non-autoregressive End-to-end Speech Translation with Parallel Autoregressive Rescoringcited by 9
  33. 2022Quantified Reproducibility Assessment of NLP Resultscited by 8
  34. 2022The Ecological Footprint of Neural Machine Translation Systemscited by 2
  35. 2022Efficient Speech Translation with Dynamic Latent Perceiverscited by 2
  36. 2021Toward Reusable Science with Readable Code and Reproducibilitycited by 1

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 6; 2025: 6.

14 papers.

Sort:

  1. 2024MOSEL: 950,000 Hours of Speech Data for Open-Source Speech Foundation Model Training on EU Languagescited by 13
  2. 2024A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Speech Translationcited by 10
  3. 2024What Do Machine Learning Researchers Mean by "Reproducible"?cited by 8
  4. 2025Voice, Bias, and Coreference: An Interpretability Study of Gender in Speech Translationcited by 5
  5. 2023No Pitch Left Behind: Addressing Gender Unbalance in Automatic Speech Recognition through Pitch Manipulationcited by 4
  6. 2025Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparisoncited by 4
  7. 2024Optimizing Rare Word Accuracy in Direct Speech Translation with a Retrieval-and-Demonstration Approachcited by 3
  8. 2025The Unheard Alternative: Contrastive Explanations for Speech-to-Text Modelscited by 3
  9. 2023How To Build Competitive Multi-gender Speech Translation Models For Controlling Speaker Gender Translationcited by 2
  10. 2025FAMA: The First Large-Scale Open-Science Speech Foundation Model for English and Italiancited by 2
  11. 2024How do Hyenas deal with Human Speech? Speech Recognition and Translation with ConfHyenacited by 1
  12. 2024Compact Speech Translation Models via Discrete Speech Units Pretrainingcited by 1
2 more
  1. 2025The Warmup Dilemma: How Learning Rate Strategies Impact Speech-to-Text Model Convergencecited by 1
  2. 2025Cross-Attention is Half Explanation in Speech-to-Text Modelscited by 1