mlsanityData to 7 Oct 2026

A Joint Spectro-Temporal Relational Thinking Based Acoustic Modeling Framework

arXiv 2409.15357 ↗First posted 17 Sep 2024eess.AS cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Relational thinking refers to the inherent ability of humans to form mental impressions about relations between sensory signals and prior knowledge, and subsequently incorporate them into their model of their world. Despite the crucial role relational thinking plays in human understanding of speech, it has yet to be leveraged in any artificial speech recognition systems. Recently, there have been some attempts to correct this oversight, but these have been limited to coarse utterance-level models that operate exclusively in the time domain. In an attempt to narrow the gap between artificial systems and human abilities, this paper presents a novel spectro-temporal relational thinking based acoustic modeling framework. Specifically, it first generates numerous probabilistic graphs to model the relationships among speech segments across both time and frequency domains. The relational information rooted in every pair of nodes within these graphs is then aggregated and embedded into latent representations that can be utilized by downstream tasks. Models built upon this framework outperform state-of-the-art systems with a 7.82% improvement in phoneme recognition tasks over the TIMIT dataset. In-depth analyses further reveal that our proposed relational thinking modeling mainly improves the model's ability to recognize vowels, which are the most likely to be confused by phoneme recognizers.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 15 of its 37 references that are in the map. First posted on arXiv 17 Sep 2024.

Builds on

15 of 37 references.

References · 15

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  4. 2015Listen, Attend and Spellcited by 542
  5. 2015A Recurrent Latent Variable Model for Sequential Datacited by 362
  6. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  7. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  8. 2019Transformer-based Acoustic Modeling for Hybrid Speech Recognitioncited by 86
  9. 2019CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognitioncited by 76
  10. 2018Light Gated Recurrent Units for Speech Recognitioncited by 48
  11. 2020GDPNet: Refining Latent Multi-View Graph for Relation Extractioncited by 29
  12. 2021Conditional Neural Relational Inference for Interacting Systemscited by 28
3 more
  1. 2017Learning Filterbanks from Raw Speech for Phone Recognitioncited by 25
  2. 2020Deep Graph Random Process for Relational-Thinking-Based Speech Recognitioncited by 12
  3. 2023Variational Connectionist Temporal Classification for Order-Preserving Sequence Modelingcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025SHANKS: Simultaneous Hearing and Thinking for Spoken Language Models
  2. 2025Thinking with Sound: Audio Chain-of-Thought Enables Multimodal Reasoning in Large Audio-Language Models
  3. 2026Speech-Hands: A Self-Reflection Voice Agentic Approach to Speech Recognition and Audio Reasoning with Omni Perception
  4. 2026AURAL: Adaptive Latent Reasoning with Joint Chunk for Speech Language Models
  5. 2025MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark
  6. 2025SpeechIQ: Speech-Agentic Intelligence Quotient Across Cognitive Levels in Voice Understanding by Large Language Models
  7. 2025SpeechR: A Benchmark for Speech Reasoning in Large Audio-Language Models
  8. 2026SciReC: Diagnostic Evaluation of Multimodal, Multi-Turn Relational Reasoning with Adaptive Interaction