# End2End Acoustic to Semantic Transduction

arXiv: https://arxiv.org/abs/2102.01013 · HTML: https://mlsanity.com/paper/2102.01013/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 30 references that are in the map.
- First posted on arXiv 1 Feb 2021.

## Builds on (most cited first, 17 of 17)

- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations](https://mlsanity.com/paper/1910.05453/) (2019)
- [End-to-End Attention-based Large Vocabulary Speech Recognition](https://mlsanity.com/paper/1508.04395/) (2015)
- [On Using Monolingual Corpora in Neural Machine Translation](https://mlsanity.com/paper/1503.03535/) (2015)
- [Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling](https://mlsanity.com/paper/1609.01454/) (2016)
- [Towards end-to-end spoken language understanding](https://mlsanity.com/paper/1802.08395/) (2018)
- [Unsupervised speech representation learning using WaveNet autoencoders](https://mlsanity.com/paper/1901.08810/) (2019)
- [End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results](https://mlsanity.com/paper/1412.1602/) (2014)
- [From Audio to Semantics: Approaches to end-to-end spoken language understanding](https://mlsanity.com/paper/1809.09190/) (2018)
- [Espresso: A Fast End-to-end Neural Speech Recognition Toolkit](https://mlsanity.com/paper/1909.08723/) (2019)
- [Curriculum-based transfer learning for an effective end-to-end spoken language understanding and domain portability](https://mlsanity.com/paper/1906.07601/) (2019)
- [End-to-end Speech Recognition with Word-based RNN Language Models](https://mlsanity.com/paper/1808.02608/) (2018)
- [Improved Speech Representations with Multi-Target Autoregressive Predictive Coding](https://mlsanity.com/paper/2004.05274/) (2020)
- [Dialogue history integration into end-to-end signal-to-concept spoken language understanding systems](https://mlsanity.com/paper/2002.06012/) (2020)
- [ASR error management for improving spoken language understanding](https://mlsanity.com/paper/1705.09515/) (2017)
- [CSTNet: Contrastive Speech Translation Network for Self-Supervised Speech Representation Learning](https://mlsanity.com/paper/2006.02814/) (2020)

## Built on by (most cited first, 10 of 10)

- [LeBenchmark 2.0: a Standardized, Replicable and Enhanced Framework for Self-supervised Representations of French Speech](https://mlsanity.com/paper/2309.05472/) (2023)
- [Where are we in semantic concept extraction for Spoken Language Understanding?](https://mlsanity.com/paper/2106.13045/) (2021)
- [Speech2Slot: An End-to-End Knowledge-based Slot Filling from Speech](https://mlsanity.com/paper/2105.04719/) (2021)
- [ASR-Generated Text for Language Model Pre-training Applied to Speech Tasks](https://mlsanity.com/paper/2207.01893/) (2022)
- [Toward Low-Cost End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2207.00352/) (2022)
- [New Semantic Task for the French Spoken Language Understanding MEDIA Benchmark](https://mlsanity.com/paper/2403.19727/) (2024)
- [Pantagruel: Unified Self-Supervised Encoders for French Text and Speech](https://mlsanity.com/paper/2601.05911/) (2026)
- [Leveraging Multilingual Self-Supervised Pretrained Models for Sequence-to-Sequence End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2310.06103/) (2023)
- [Is one brick enough to break the wall of spoken dialogue state tracking?](https://mlsanity.com/paper/2311.04923/) (2023)
- [Vers la compr'ehension automatique de la parole bout-en-bout `a moindre effort](https://mlsanity.com/paper/2207.00349/) (2022)

## Abstract (arXiv)

In this paper, we propose a novel end-to-end sequence-to-sequence spoken language understanding model using an attention mechanism. It reliably selects contextual acoustic features in order to hypothesize semantic contents. An initial architecture capable of extracting all pronounced words and concepts from acoustic spans is designed and tested. With a shallow fusion language model, this system reaches a 13.6 concept error rate (CER) and an 18.5 concept value error rate (CVER) on the French MEDIA corpus, achieving an absolute 2.8 points reduction compared to the state-of-the-art. Then, an original model is proposed for hypothesizing concepts and their values. This transduction reaches a 15.4 CER and a 21.6 CVER without any new type of context.