mlsanityData to 19 Oct 2026

Hearing's Blind Spots: Toward Perceptual Compute Allocation in Neural Text-to-Speech

arXiv 2610.14056 ↗First posted 6 Oct 2026cs.SD eess.AScited by 0

On the map: Speech and audio language models → Vocoders and speech synthesis also fits: Speech enhancement and noise suppression

Abstract (arXiv)

A speech synthesizer produces a detailed waveform, but a listener need not hear every detail. Strong sounds can mask weaker neighbors, creating context-dependent "blind spots" in hearing. Could neural text-to-speech (TTS) spend less computation on those details? We investigate a first step: how the placement of spectral degradation affects completed speech from Piper and Kokoro. At matched time-frequency bin-removal budgets, a simple low-energy-first policy achieves higher mean wideband PESQ than random deletion and the tested psychoacoustic masking criteria. At 50% removal in the spreading-matrix comparison, energy, masking, and random policies score 4.46, 3.60, and 1.19 on Piper, and 3.42, 2.10, and 1.22 on Kokoro. Spectrograms and masking maps expose where these perturbations fall and why a model's below-threshold regions cannot automatically be treated as inaudible. Calibration diagnostics show substantial threshold sensitivity, while a small single-listener pilot establishes no transparent removal budget. CPU profiles identify waveform decoding as the main measured kernel cost, but a tested dynamic-INT8 Kokoro decoder runs slower than FP32 on both CPUs. The results establish energy ranking as a strong oracle baseline and motivate testing perceptual allocation inside a decoder. They do not yet demonstrate saved computation: the decisions here use already synthesized audio, and quality metrics are evaluated at 16 kHz. We release scripts, results, audio examples, and a listening-study instrument.

Key facts. No citing papers in the map yet (data through 19 Oct 2026). Placed in Vocoders and speech synthesis, one of 11 directions in Speech and audio language models. Builds on 21 of its 45 references that are in the map. First posted on arXiv 6 Oct 2026.

Builds on

21 of 45 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Robust and Efficient Autoregressive Speech Synthesis with Dynamic Chunk-wise Prediction Policy27 Jun
  2. 2025Accelerating Flow-Matching-Based Text-to-Speech via Empirically Pruned Step Sampling26 May
  3. 2025Accelerating Autoregressive Speech Synthesis Inference With Speech Speculative Decoding21 May
  4. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models13 Jun
  5. 2022iSTFTNet: Fast and Lightweight Mel-Spectrogram Vocoder Incorporating Inverse Short-Time Fourier Transform4 Mar
  6. 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech11 Jun

Other references · 15

  1. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,691
  2. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,462
  3. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,050
  4. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 947
  5. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 602
  6. 2021Dynamic Neural Networks: A Surveycited by 416
  7. 2022Confident Adaptive Language Modelingcited by 319
  8. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 271
  9. 2018LPCNet: Improving Neural Speech Synthesis Through Linear Predictioncited by 160
  10. 2020Psychoacoustic Calibration of Loss Functions for Efficient End-to-End Neural Audio Codingcited by 9
  11. 2025Multi-band Frequency Reconstruction for Neural Psychoacoustic Codingcited by 7
  12. 2024DAISY: Data Adaptive Self-Supervised Early Exit for Speech Representation Modelscited by 5
3 more
  1. 2026Foveated Diffusion: Efficient Spatially Adaptive Image and Video Generationcited by 5
  2. 2018On Psychoacoustically Weighted Cost Functions Towards Resource-Efficient Deep Neural Networks for Speech Denoisingcited by 1
  3. 2026SPAR-K: Scheduled Periodic Alternating Early Exit for Spoken Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025From Evaluation to Optimization: Neural Speech Assessment for Downstream Applications ↗
  2. 2026Probing Low Frame Rate Degradation in Neural Audio Codecs
  3. 2026Where Does the Sound Go? Tracing Acoustic Information Loss in Audio-Conditioned LLMs
  4. 2026Encoder-Side Neuron Identification and Amplification for Acoustic Perception in Large Audio-Language Models
  5. 2026Perceptually Better, Semantically Worse: Measuring Speech Enhancement Impact on LLM-Based Voice Systems
  6. 2025ISA-Bench: Benchmarking Instruction Sensitivity for Large Audio Language Models
  7. 2026From Inaudible Inputs to Model Failures: Low-Frequency Safety Risks in LALMs
  8. 2026Adaptive Perturbation Selection for Contrastive Audio Decoding