mlsanityData to 7 Oct 2026

Quantitative Analysis of Audio-Visual Tasks: An Information-Theoretic Perspective

arXiv 2409.19575 ↗First posted 29 Sep 2024cs.SD cs.CL cs.MMcited by 0

Not on the map. Why?

Abstract (arXiv)

In the field of spoken language processing, audio-visual speech processing is receiving increasing research attention. Key components of this research include tasks such as lip reading, audio-visual speech recognition, and visual-to-speech synthesis. Although significant success has been achieved, theoretical analysis is still insufficient for audio-visual tasks. This paper presents a quantitative analysis based on information theory, focusing on information intersection between different modalities. Our results show that this analysis is valuable for understanding the difficulties of audio-visual processing tasks as well as the benefits that could be obtained by modality integration.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 15 of its 32 references that are in the map. First posted on arXiv 29 Sep 2024.

Builds on

15 of 32 references.

References · 15

  1. 2022Generative Adversarial Networkscited by 5,603
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  3. 2018Deep Audio-Visual Speech Recognitioncited by 318
  4. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  5. 2016Lip Reading Sentences in the Wildcited by 226
  6. 2021WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognitioncited by 189
  7. 2018Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separationcited by 160
  8. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 113
  9. 2016LipNet: End-to-End Sentence-level Lipreadingcited by 79
  10. 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 72
  11. 2021VQMIVC: Vector Quantization and Mutual Information-Based Unsupervised Speech Representation Disentanglement for One-shot Voice Conversioncited by 63
  12. 2022SVTS: Scalable Video-to-Speech Synthesiscited by 35
3 more
  1. 2024Revisiting Self-supervised Learning of Speech Representation from a Mutual Information Perspectivecited by 5
  2. 2024CNVSRC 2023: The First Chinese Continuous Visual Speech Recognition Challengecited by 5
  3. 2022The impact of removing head movements on audio-visual speech enhancementcited by 3

Built on by

Not cited yet.