mlsanityData to 14 Oct 2026

Long-Term Memory-Guided Enhancement for Target Perception in Audio-Language Models

arXiv 2609.36577 ↗First posted 29 Sep 2026cs.SD cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Audio large language models (ALLMs) can reason about the content of audio recordings to perform complex tasks. However, these capabilities usually collapse in real-world environments when background noise and competing sources mix the target sound. Inspired by long-term memory in human listening, we propose Long-Term Memory-Guided Audio Enhancement (LTM-AE) to improve selective target perception by refining the audio representations of ALLMs without training. LTM-AE extracts representations in hidden states from separate clean reference recordings as long-term memory for each category, guiding enhancement toward a user-specified listening target. We reconstruct incoming audio tokens in the selected category long-term memory and interpolate the reconstructions with the original tokens before language backbone decoding. This interpolation controls the influence of stored auditory experience while keeping all ALLM parameters fixed. Diagnostic readouts across twenty sound categories and three ALLMs show that LTM-AE strengthens responses to a specified target amid three interfering sources. Averaged over constrained and free-form classification, accuracy gains over raw mixtures range from 29.53 to 46.15 percentage points across multiple open source models. For speech content recovery, LTM-AE with an additional learned token-level gate reduces Qwen2-Audio's word error rate from 23.07% to 14.77%. This work takes an initial step toward using principles of human long-term memory to enhance ALLMs for real-world listening. Our code is available at https://github.com/aynlp/ltm-audio-code

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 37 of its 65 references that are in the map. First posted on arXiv 29 Sep 2026.

Builds on

37 of 65 references.

References · 37

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  3. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  4. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  5. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  6. 2024Qwen2-Audio Technical Reportcited by 560
  7. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 556
  8. 2018Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separationcited by 406
  9. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 342
  10. 2022Memorizing Transformerscited by 226
  11. 2017Neural Audio Synthesis of Musical Notes with WaveNet Autoencoderscited by 225
  12. 2025Kimi-Audio Technical Reportcited by 222
25 more
  1. 2018Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Datasetcited by 217
  2. 2017Supervised Speech Separation Based on Deep Learning: An Overviewcited by 216
  3. 2022Speech Enhancement and Dereverberation with Diffusion-based Generative Modelscited by 197
  4. 2017SEGAN: Speech Enhancement Generative Adversarial Networkcited by 166
  5. 2020DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancementcited by 141
  6. 2025Step-Audio 2 Technical Reportcited by 105
  7. 2022Separate What You Describe: Language-Queried Audio Source Separationcited by 45
  8. 2023Separate Anything You Describecited by 43
  9. 2022Text-Driven Separation of Arbitrary Soundscited by 23
  10. 2025VoxEval: Benchmarking the Knowledge Understanding Capabilities of End-to-End Spoken Language Modelscited by 16
  11. 2025AudioTrust: Benchmarking the Multifaceted Trustworthiness of Audio Large Language Modelscited by 14
  12. 2025Evaluating Robustness of Large Audio Language Models to Audio Injection: An Empirical Studycited by 12
  13. 2025Self-Improvement for Audio Large Language Model using Unlabeled Speechcited by 10
  14. 2024CLAPSep: Leveraging Contrastive Pre-trained Model for Multi-Modal Query-Conditioned Target Sound Extractioncited by 9
  15. 2025Thinking with Sound: Audio Chain-of-Thought Enables Multimodal Reasoning in Large Audio-Language Modelscited by 9
  16. 2024GAPS: A Large and Diverse Classical Guitar Dataset and Benchmark Transcription Modelcited by 8
  17. 2025Can Large Audio Language Models Understand Audio Well? Speech, Scene and Events Understanding Benchmark for LALMscited by 5
  18. 2025AV-Dialog: Spoken Dialogue Models with Audio-Visual Inputcited by 5
  19. 2026RSA-Bench: Benchmarking Audio Large Models in Real-World Acoustic Scenarioscited by 5
  20. 2025VCB Bench: An Evaluation Benchmark for Audio-Grounded Large Language Model Conversational Agentscited by 4
  21. 2026SEE: Signal Embedding Energy for Quantifying Noise Interference in Large Audio Language Modelscited by 3
  22. 2026Focus Then Listen: An Empirical Study of Plug-and-Play Audio Enhancer for Noise-Robust Large Audio Language Modelscited by 3
  23. 2026SCENEBench: An Audio Understanding Benchmark Grounded in Assistive and Industrial Use Casescited by 3
  24. 2026Listen, Pause, and Reason: Toward Perception-Grounded Hybrid Reasoning for Audio Understandingcited by 2
  25. 2026Noise Adaptive Streaming Audio-Visual Speech Token Enhancement for Robust Full-Duplex Spoken Dialogue Modelscited by 1

Built on by

Not cited yet.