mlsanityData to 7 Oct 2026

Inside the Latent Flow: Causal Deciphering of Attention Dynamics in Audio Separation Foundation Models

arXiv 2606.10046 ↗First posted 8 Jun 2026cs.SD cs.AIcited by 0

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Flow-matching transformers achieve strong audio separation, yet their attention dynamics are opaque. We adapt established causal-intervention principles into a deterministic, inference-time probing protocol for SAM Audio. Orthogonal probing uncovers a dual-pathway text-conditioning mechanism: additive injections control semantic identity, while cross-attention refines acoustic structure. We observe an asynchronous layerwise convergence: stable layers build temporal scaffolds early, whereas fast layers continue resolving artifacts during sampling. The model also attenuates temporal segmentation cues to maintain continuous-flow stability. Using these insights, we propose Layer-Selective Attention Caching (LSAC), a training-free acceleration method that caches attention in stable layers. Across acoustic complexities, LSAC cuts self-attention computation by about ~25% with negligible quality loss and yields up to 6.7x higher quality retention than naive step reduction.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 38 of its 51 references that are in the map. First posted on arXiv 8 Jun 2026.

Builds on

38 of 51 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025SAM Audio: Segment Anything in Audio19 Dec
  2. 2025MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extraction29 May
  3. 2024EzAudio: Enhancing Text-to-Audio Generation with Efficient Diffusion Transformer17 Sep
  4. 2024FlowSep: Language-Queried Sound Separation with Rectified Flow Matching11 Sep
  5. 2024LiteFocus: Accelerated Diffusion Inference for Long Audio Synthesis15 Jul
  6. 2024Complex Image-Generative Diffusion Transformer for Audio Denoising13 Jun
  7. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining10 Aug

Other references · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2023Segment Anythingcited by 8,158
  7. 2022Scalable Diffusion Models with Transformerscited by 5,931
  8. 2022Flow Matching for Generative Modelingcited by 5,426
  9. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  10. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  11. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  12. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,961
19 more
  1. 2019Attention is not Explanationcited by 916
  2. 2020Quantifying Attention Flow in Transformerscited by 759
  3. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 696
  4. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  5. 2023High-Fidelity Audio Compression with Improved RVQGANcited by 423
  6. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 400
  7. 2023DeepCache: Accelerating Diffusion Models for Freecited by 346
  8. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 339
  9. 2022What the DAAM: Interpreting Stable Diffusion Using Cross Attentioncited by 233
  10. 2024Stable Audio Opencited by 192
  11. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 178
  12. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 174
  13. 2024Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editingcited by 134
  14. 2022SepTr: Separable Transformer for Audio Spectrogram Processingcited by 15
  15. 2023Masked-Attention Diffusion Guidance for Spatially Controlling Text-to-Image Generationcited by 9
  16. 2024Causal Deciphering and Inpainting in Spatio-Temporal Dynamics via Diffusion Modelcited by 6
  17. 2023Ripple sparse self-attention for monaural speech enhancementcited by 3
  18. 2024Disentangled-Transformer: An Explainable End-to-End Automatic Speech Recognition Model with Speech Content-Context Separationcited by 1
  19. 2025Time-Frequency-Based Attention Cache Memory Model for Real-Time Speech Separationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Models
  2. 2026A Closer Look at Failure Modes in Temporal Understanding of Large Audio-Language Models
  3. 2026SoundWeaver: Semantic Warm-Starting for Text-to-Audio Diffusion Serving
  4. 2026WnW: Waxing-and-Waning KV Cache for Long-Form Speech LLMs
  5. 2025AIBA: Attention-based Instrument Band Alignment for Text-to-Audio Diffusion
  6. 2025StreamFlow: Streaming Flow Matching with Block-wise Guided Attention Mask for Speech Token Decoding
  7. 2026TADA! Tuning Audio Diffusion Models through Activation Steering
  8. 2026CAWN: Continuous Acoustic Wave Networks for Autoregressive Language Modeling