# Sequence-to-sequence models in peer-to-peer learning: A practical application

arXiv: https://arxiv.org/abs/2406.02565 · HTML: https://mlsanity.com/paper/2406.02565/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 28 references that are in the map.
- First posted on arXiv 2 May 2024.

## Builds on (most cited first, 17 of 17)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [FedBN: Federated Learning on Non-IID Features via Local Batch Normalization](https://mlsanity.com/paper/2102.07623/) (2021)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [Stochastic Gradient Push for Distributed Deep Learning](https://mlsanity.com/paper/1811.10792/) (2018)
- [Hybrid Autoregressive Transducer (hat)](https://mlsanity.com/paper/2003.07705/) (2020)
- [Multi-Task Federated Learning for Personalised Deep Neural Networks in Edge Computing](https://mlsanity.com/paper/2007.09236/) (2020)
- [Neural Abstractive Text Summarization with Sequence-to-Sequence Models](https://mlsanity.com/paper/1812.02303/) (2018)
- [Gossip training for deep learning](https://mlsanity.com/paper/1611.09726/) (2016)
- [Towards Effective Clustered Federated Learning: A Peer-to-peer Framework with Adaptive Neighbor Matching](https://mlsanity.com/paper/2203.12285/) (2022)
- [How to scale distributed deep learning?](https://mlsanity.com/paper/1611.04581/) (2016)
- [Federated Learning for ASR based on Wav2vec 2.0](https://mlsanity.com/paper/2302.10790/) (2023)
- [Unsupervised pre-training for sequence to sequence speech recognition](https://mlsanity.com/paper/1910.12418/) (2019)
- [UserLibri: A Dataset for ASR Personalization Using Only Text](https://mlsanity.com/paper/2207.00706/) (2022)
- [Federated Natural Language Generation for Personalized Dialogue System](https://mlsanity.com/paper/2110.06419/) (2021)
- [Synthesizing Speech Test Cases with Text-to-Speech? An Empirical Study on the False Alarms in Automated Speech Recognition Testing](https://mlsanity.com/paper/2305.17445/) (2023)

## Abstract (arXiv)

This paper explores the applicability of sequence-to-sequence (Seq2Seq) models based on LSTM units for Automatic Speech Recognition (ASR) task within peer-to-peer learning environments. Leveraging two distinct peer-to-peer learning methods, the study simulates the learning process of agents and evaluates their performance in ASR task using two different ASR datasets. In a centralized training setting, utilizing a scaled-down variant of the Deep Speech 2 model, a single model achieved a Word Error Rate (WER) of 84% when trained on the UserLibri dataset, and 38% when trained on the LJ Speech dataset. Conversely, in a peer-to-peer learning scenario involving 55 agents, the WER ranged from 87% to 92% for the UserLibri dataset, and from 52% to 56% for the LJ Speech dataset. The findings demonstrate the feasibility of employing Seq2Seq models in decentralized settings, albeit with slightly higher Word Error Rates (WER) compared to centralized training methods.