# CoSTALA: Compositional Spatio-Temporal Audio-Language Alignment via Multi-Grain Hierarchical Contrastive Learning

arXiv: https://arxiv.org/abs/2608.24374 · HTML: https://mlsanity.com/paper/2608.24374/

## Key facts

- Cited by 1 papers in the map (data through 19 Oct 2026).
- Placed in Audio language models and spoken dialogue, one of 11 directions in Speech and audio language models.
- Builds on 26 of its 27 references that are in the map.
- First posted on arXiv 25 Aug 2026.

## Placed in

- Speech and audio language models / [Audio language models and spoken dialogue](https://mlsanity.com/speech/audio-language-models-and-spoken-dialogue/)

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection](https://mlsanity.com/paper/2202.00874/) (2022)
- [BAT: Learning to Reason about Spatial Sounds with Large Language Models](https://mlsanity.com/paper/2402.01591/) (2024)
- [An Improved Event-Independent Network for Polyphonic Sound Event Localization and Detection](https://mlsanity.com/paper/2010.13092/) (2020)
- [T-CLAP: Temporal-Enhanced Contrastive Language-Audio Pretraining](https://mlsanity.com/paper/2404.17806/) (2024)
- [Learning Spatially-Aware Language and Audio Embeddings](https://mlsanity.com/paper/2409.11369/) (2024)
- [Advancing Multi-grained Alignment for Contrastive Language-Audio Pre-training](https://mlsanity.com/paper/2408.07919/) (2024)
- [Can Large Language Models Understand Spatial Audio?](https://mlsanity.com/paper/2406.07914/) (2024)
- [DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioning](https://mlsanity.com/paper/2410.09472/) (2024)
- [M2D-CLAP: Exploring General-purpose Audio-Language Representations Beyond CLAP](https://mlsanity.com/paper/2503.22104/) (2025)
- [SPUR: A Plug-and-Play Framework for Integrating Spatial Audio Understanding and Reasoning into Large Audio-Language Models](https://mlsanity.com/paper/2511.06606/) (2025)
- [SALM: Spatial Audio Language Model with Structured Embeddings for Understanding and Editing](https://mlsanity.com/paper/2507.16724/) (2025)
- [PSELDNets: Pre-trained Neural Networks on a Large-scale Synthetic Dataset for Sound Event Localization and Detection](https://mlsanity.com/paper/2411.06399/) (2024)
- [DSpAST: Disentangled Representations for Spatial Audio Reasoning with Large Language Models](https://mlsanity.com/paper/2509.13927/) (2025)
- [Enhance Temporal Relations in Audio Captioning with Sound Event Detection](https://mlsanity.com/paper/2306.01533/) (2023)
- [Text-Queried Target Sound Event Localization](https://mlsanity.com/paper/2406.16058/) (2024)
- [Towards Spatial Audio Understanding via Question Answering](https://mlsanity.com/paper/2507.09195/) (2025)
- [Spatial-CLAP: Learning Spatially-Aware audio--text Embeddings for Multi-Source Conditions](https://mlsanity.com/paper/2509.14785/) (2025)
- [Towards Multimodal Query-Based Spatial Audio Source Extraction](https://mlsanity.com/paper/2510.13308/) (2025)

## Built on by (most cited first, 1 of 1)

- [MiDashengLM-Spatial: Unifying General Audio Understanding and Spatial Awareness](https://mlsanity.com/paper/2610.11156/) (2026)

## Abstract (arXiv)

Conventional audio language models (ALMs) have made significant progress in achieving alignment between auditory and textual representations, including recent explorations in spatial audio. However, in daily spatial scenarios, they still cannot effectively process multi-event audio sequences. Current approaches primarily rely on coarse-grained contrastive learning with global auditory and textual features, lacking the resolution to distinguish multiple sequential events. To overcome these limitations, we propose CoSTALA-a novel training paradigm that transitions from purely global alignment to fine-grained spatio-temporal reasoning. By constructing a multi-granularity hierarchical loss function system, we achieve explicit modeling of temporal dependencies, and successfully anchors individual acoustic events to preserve their semantic purity. Extensive experiments demonstrate that CoSTALA significantly establish a powerful new framework for spatio-temporal audio understanding.