mlsanityData to 7 Oct 2026

Open-Source Conversational AI with SpeechBrain 1.0

arXiv 2407.00463 ↗First posted 29 Jun 2024cs.LG cs.AI cs.CLcited by 55

Not on the map. Why?

Abstract (arXiv)

SpeechBrain is an open-source Conversational AI toolkit based on PyTorch, focused particularly on speech processing tasks such as speech recognition, speech enhancement, speaker recognition, text-to-speech, and much more. It promotes transparency and replicability by releasing both the pre-trained models and the complete "recipes" of code and algorithms required for training them. This paper presents SpeechBrain 1.0, a significant milestone in the evolution of the toolkit, which now has over 200 recipes for speech, audio, and language processing tasks, and more than 100 models available on Hugging Face. SpeechBrain 1.0 introduces new technologies to support diverse learning modalities, Large Language Model (LLM) integration, and advanced decoding strategies, along with novel models, tasks, and modalities. It also includes a new benchmark repository, offering researchers a unified platform for evaluating models across diverse tasks.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 39 of its 53 references that are in the map. First posted on arXiv 29 Jun 2024.

Builds on

39 of 53 references.

References · 39

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  4. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,339
  5. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  6. 2017Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictionscited by 800
  7. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  8. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  9. 2016EEGNet: A Compact Convolutional Network for EEG-based Brain-Computer Interfacescited by 616
  10. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 606
  11. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  12. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
27 more
  1. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 400
  2. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 254
  3. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 230
  4. 2019Dual-path RNN: efficient long sequence modeling for time-domain single-channel speech separationcited by 182
  5. 2023Joint Audio and Speech Understandingcited by 99
  6. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 90
  7. 2018Light Gated Recurrent Units for Speech Recognitioncited by 48
  8. 2024DASB - Discrete Audio and Speech Benchmarkcited by 32
  9. 2023Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?cited by 28
  10. 2023TorchAudio 2.1: Advancing speech recognition, self-supervised learning, and audio processing components for PyTorchcited by 25
  11. 2024How Should We Extract Discrete Audio Tokens from Self-Supervised Models?cited by 24
  12. 2022Listen to Interpret: Post-hoc Interpretability for Audio Networks with NMFcited by 23
  13. 2019RNN-T For Latency Controlled ASR With Improved Beam Searchcited by 19
  14. 2023LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speechcited by 17
  15. 2022SkiM: Skipping Memory LSTM for Low-Latency Real-Time Continuous Speech Separationcited by 16
  16. 2023CL-MASR: A Continual Learning Benchmark for Multilingual ASRcited by 16
  17. 2023Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models: A Comparative Studycited by 15
  18. 2023Speech Emotion Diarization: Which Emotion Appears When?cited by 13
  19. 2024Listenable Maps for Audio Classifierscited by 13
  20. 2022Fast and parallel decoding for transducercited by 12
  21. 2024Open Implementation and Study of BEST-RQ for Speech Processingcited by 9
  22. 2024Focal Modulation Networks for Interpretable Sound Classificationcited by 8
  23. 2023Posthoc Interpretation via Quantizationcited by 7
  24. 2024Are LLMs Robust for Spoken Dialogues?cited by 7
  25. 2017Contaminated speech training methods for robust DNN-HMM distant speech recognitioncited by 5
  26. 2023HyperConformer: Multi-head HyperMixer for Efficient Speech Recognitioncited by 5
  27. 2023Stabilising and accelerating light gated recurrent units for automatic speech recognitioncited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 32; 2026: 19.

55 papers.

Sort:

By area:

  1. 2025Paper2Video: Automatic Video Generation from Scientific Paperscited by 23
  2. 2025Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluationcited by 21
  3. 2025FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networkscited by 18
  4. 2025Loquacious Set: 25,000 Hours of Transcribed and Diverse English Speech Recognition Data for Research and Commercial Usecited by 9
  5. 2025Evaluation of LLMs in Speech is Often Flawed: Test Set Contamination in Large Language Models for Speech Recognitioncited by 9
  6. 2026Do Audio LLMs Listen or Read? Analyzing and Mitigating Paralinguistic Failures with VoxParadoxcited by 7
  7. 2025MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Taskscited by 6
  8. 2025SENSE models: an open source solution for multilingual and multimodal semantic-based taskscited by 6
  9. 2025SpidR: Learning Fast and Stable Linguistic Units for Spoken Language Models Without Supervisioncited by 6
  10. 2024LMAC-TD: Producing Time Domain Explanations for Audio Classifierscited by 4
  11. 2024Investigating the Effectiveness of Explainability Methods in Parkinson's Detection from Speechcited by 4
  12. 2024Open Universal Arabic ASR Leaderboardcited by 4
38 more
  1. 2025Voice Cloning: Comprehensive Surveycited by 4
  2. 2025A Study of the Scale Invariant Signal to Distortion Ratio in Speech Separation with Noisy Referencescited by 4
  3. 2025Real-Time Streamable Generative Speech Restoration with Flow Matchingcited by 4
  4. 2024Lightweight Model Attribution and Detection of Synthetic Speech via Audio Residual Fingerprintscited by 3
  5. 2025Zero-Shot Text-to-Speech for Vietnamesecited by 3
  6. 2025Marco-Voice Technical Reportcited by 3
  7. 2025ParsVoice: A Large-Scale Multi-Speaker Persian Speech Corpus for Text-to-Speech Synthesiscited by 3
  8. 2025ADI-20: Arabic Dialect Identification dataset and modelscited by 3
  9. 2025TEDxTN: A Three-way Speech Translation Corpus for Code-Switched Tunisian Arabic - Englishcited by 3
  10. 2025Toward Conversational Hungarian Speech Recognition: Introducing the BEA-Large and BEA-Dialogue Datasetscited by 3
  11. 2026Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenizationcited by 3
  12. 2025Scaling Spoken Language Models with Syllabic Speech Tokenizationcited by 2
  13. 2025Benchmarking Rotary Position Embeddings for Automatic Speech Recognitioncited by 1
  14. 2025Steganography Beyond Space-Time with Chain of Multimodal AIcited by 1
  15. 2025You Are What You Say: Exploiting Linguistic Content for VoicePrivacy Attackscited by 1
  16. 2025Autoregressive Speech Enhancement via Acoustic Tokenscited by 1
  17. 2025From Black Box to Biomarker: Sparse Autoencoders for Interpreting Speech Models of Parkinson's Diseasecited by 1
  18. 2025In-domain SSL pre-training and streaming ASRcited by 1
  19. 2025Real-Time Streaming Mel Vocoding with Generative Flow Matchingcited by 1
  20. 2025The Speech-LLM Takes It All: A Truly Fully End-to-End Spoken Dialogue State Tracking Approachcited by 1
  21. 2026MK-SGC-SC: Multiple Kernel Guided Sparse Graph Construction in Spectral Clustering for Unsupervised Speaker Diarizationcited by 1
  22. 2026Disentangling Speaker Traits for Deepfake Source Verification via Chebyshev Polynomial and Riemannian Metric Learningcited by 1
  23. 2026Ara-Best-RQ: Multi Dialectal Arabic SSLcited by 1
  24. 2025Active Speech Enhancement: Active Speech Denoising Decliping and Deveraberationcited by 0
  25. 2025Robust Unsupervised Adaptation of a Speech Recogniser Using Entropy Minimisation and Speaker Codescited by 0
  26. 2025Task-Specific Audio Coding for Machines: Machine-Learned Latent Features Are Codes for That Machinecited by 0
  27. 2025Evaluating and Improving the Robustness of Speech Command Recognition Models to Noise and Distribution Shiftscited by 0
  28. 2025Enhancing Speech Emotion Recognition via Fine-Tuning Pre-Trained Models and Hyper-Parameter Optimisationcited by 0
  29. 2025DroneAudioset: An Audio Dataset for Drone-based Search and Rescuecited by 0
  30. 2026TCG CREST System Description for the DISPLACE-M Challengecited by 0
  31. 2026Measuring the Redundancy of Decoder Layers in SpeechLLMscited by 0
  32. 2026Using Multimodal and Language-Agnostic Sentence Embeddings for Abstractive Summarizationcited by 0
  33. 2026Learning Multiple Utterance-Level Attribute Representations with a Unified Speech Encodercited by 0
  34. 2026SLURP-TN : Resource for Tunisian Dialect Spoken Language Understandingcited by 0
  35. 2026CV-18 NER: Augmented Common Voice for Named Entity Recognition from Arabic Speechcited by 0
  36. 2026DiffAnon: Diffusion-based Prosody Control for Voice Anonymizationcited by 0
  37. 2026Streaming Speech-to-Text Translation with a SpeechLLMcited by 0
  38. 2026MambAdapter: Lightweight Mamba-Based Adapters for Parameter-Efficient Transfer Learning in Speech and Audiocited by 0

50 most cited of 55 All 55 (JSON)