mlsanityData to 14 Oct 2026

DiffNorm: Self-Supervised Normalization for Non-autoregressive Speech-to-speech Translation

arXiv 2405.13274 ↗First posted 22 May 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Non-autoregressive Transformers (NATs) are recently applied in direct speech-to-speech translation systems, which convert speech across different languages without intermediate text data. Although NATs generate high-quality outputs and offer faster inference than autoregressive models, they tend to produce incoherent and repetitive results due to complex data distribution (e.g., acoustic and linguistic variations in speech). In this work, we introduce DiffNorm, a diffusion-based normalization strategy that simplifies data distributions for training NAT models. After training with a self-supervised noise estimation objective, DiffNorm constructs normalized target data by denoising synthetically corrupted speech features. Additionally, we propose to regularize NATs with classifier-free guidance, improving model robustness and translation quality by randomly dropping out source information during training. Our strategies result in a notable improvement of about +7 ASR-BLEU for English-Spanish (En-Es) and +2 ASR-BLEU for English-French (En-Fr) translations on the CVSS benchmark, while attaining over 14x speedup for En-Es and 5x speedup for En-Fr translations compared to autoregressive baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 45 of its 60 references that are in the map. First posted on arXiv 22 May 2024.

Builds on

45 of 60 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  10. 2022Scalable Diffusion Models with Transformerscited by 6,054
  11. 2022Classifier-Free Diffusion Guidancecited by 5,847
  12. 2022Generative Adversarial Networkscited by 5,627
33 more
  1. 2017Neural Discrete Representation Learningcited by 4,712
  2. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2021Improved Denoising Diffusion Probabilistic Modelscited by 3,761
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  8. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  9. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  10. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,163
  11. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  12. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 804
  13. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  14. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 599
  15. 2017Tacotron: Towards End-to-End Speech Synthesiscited by 490
  16. 2017Non-Autoregressive Neural Machine Translationcited by 463
  17. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  18. 2021Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speechcited by 447
  19. 2021Speech Resynthesis from Discrete Disentangled Self-Supervised Representationscited by 243
  20. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 235
  21. 2019Levenshtein Transformercited by 199
  22. 2019Direct speech-to-speech translation with a sequence-to-sequence modelcited by 118
  23. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 80
  24. 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
  25. 2022CVSS Corpus and Massively Multilingual Speech-to-Speech Translationcited by 71
  26. 2022UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Unitscited by 60
  27. 2022HiFi++: a Unified Framework for Bandwidth Extension and Speech Enhancementcited by 49
  28. 2022Directed Acyclic Transformer for Non-Autoregressive Machine Translationcited by 44
  29. 2020UWSpeech: Speech to Speech Translation for Unwritten Languagescited by 43
  30. 2022TranSpeech: Speech-to-Speech Translation With Bilateral Perturbationcited by 34
  31. 2019Speech-to-speech Translation between Untranscribed Unknown Languagescited by 27
  32. 2022Speech-to-Speech Translation For A Real-world Unwritten Languagecited by 25
  33. 2023DASpeech: Directed Acyclic Transformer for Fast and High-quality Speech-to-Speech Translationcited by 9

Built on by

Not cited yet.