# DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignment

arXiv: https://arxiv.org/abs/2507.02768 · HTML: https://mlsanity.com/paper/2507.02768/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- Placed in Self-rewarding alignment and self-training, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 103 of its 132 references that are in the map.
- First posted on arXiv 3 Jul 2025.

## Placed in

- Synthetic data and self-improvement for LLMs / [Self-rewarding alignment and self-training](https://mlsanity.com/synth/self-rewarding-alignment-and-self-training/)

## Builds on (most cited first, 30 of 103)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Gemma 3 Technical Report](https://mlsanity.com/paper/2503.19786/) (2025)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [VoxCeleb: a large-scale speaker identification dataset](https://mlsanity.com/paper/1706.08612/) (2017)
- [An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks](https://mlsanity.com/paper/1312.6211/) (2013)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [VILA: On Pre-training for Visual Language Models](https://mlsanity.com/paper/2312.07533/) (2023)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model](https://mlsanity.com/paper/2304.15010/) (2023)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [Otter: A Multi-Modal Model with In-Context Instruction Tuning](https://mlsanity.com/paper/2305.03726/) (2023)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://mlsanity.com/paper/2503.01743/) (2025)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [MUSAN: A Music, Speech, and Noise Corpus](https://mlsanity.com/paper/1510.08484/) (2015)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)

## Built on by (most cited first, 30 of 50)

- [STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Models](https://mlsanity.com/paper/2507.15375/) (2025)
- [X-OPD: Cross-Modal On-Policy Distillation for Capability Alignment in Speech LLMs](https://mlsanity.com/paper/2603.24596/) (2026)
- [EchoMind: An Interrelated Multi-level Benchmark for Evaluating Empathetic Speech Language Models](https://mlsanity.com/paper/2510.22758/) (2025)
- [STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence](https://mlsanity.com/paper/2510.24693/) (2025)
- [UALM: Unified Audio Language Model for Understanding, Generation and Reasoning](https://mlsanity.com/paper/2510.12000/) (2025)
- [Closing the Modality Reasoning Gap for Speech Large Language Models](https://mlsanity.com/paper/2601.05543/) (2026)
- [All That Glitters Is Not Audio: Rethinking Text Priors and Audio Reliance in Audio-Language Evaluation](https://mlsanity.com/paper/2604.24401/) (2026)
- [Cross-Modal Knowledge Distillation for Speech Large Language Models](https://mlsanity.com/paper/2509.14930/) (2025)
- [How Contrastive Decoding Enhances Large Audio Language Models](https://mlsanity.com/paper/2603.09232/) (2026)
- [Hearing the Order: Investigating Position Bias in Large Audio-Language Models](https://mlsanity.com/paper/2510.00628/) (2025)
- [SAKE: Towards Editing Auditory Attribute Knowledge of Large Audio-Language Models](https://mlsanity.com/paper/2510.16917/) (2025)
- [Causal Tracing of Audio-Text Fusion in Large Audio Language Models](https://mlsanity.com/paper/2603.13768/) (2026)
- [When Silence Matters: The Impact of Irrelevant Audio on Text Reasoning in Large Audio-Language Models](https://mlsanity.com/paper/2510.00626/) (2025)
- [Investigating Safety Vulnerabilities of Large Audio-Language Models Under Speaker Emotional Variations](https://mlsanity.com/paper/2510.16893/) (2025)
- [Bagpiper: Solving Open-Ended Audio Tasks via Rich Captions](https://mlsanity.com/paper/2602.05220/) (2026)
- [ISA-Bench: Benchmarking Instruction Sensitivity for Large Audio Language Models](https://mlsanity.com/paper/2510.23558/) (2025)
- [CantoASR: Prosody-Aware ASR-LALM Collaboration for Low-Resource Cantonese](https://mlsanity.com/paper/2511.04139/) (2025)
- [AzeroS: Extending LLM to Speech with Self-Generated Instruction-Free Tuning](https://mlsanity.com/paper/2601.06086/) (2025)
- [DIFFA-2: A Practical Diffusion Large Language Model for General Audio Understanding](https://mlsanity.com/paper/2601.23161/) (2026)
- [Echo: Towards Advanced Audio Comprehension via Audio-Interleaved Reasoning](https://mlsanity.com/paper/2602.11909/) (2026)
- [ASPIRin: Action Space Projection for Interactivity-Optimized Reinforcement Learning in Full-Duplex Speech Language Models](https://mlsanity.com/paper/2604.10065/) (2026)
- [VIBE: Voice-Induced open-ended Bias Evaluation for Large Audio-Language Models via Real-World Speech](https://mlsanity.com/paper/2604.17248/) (2026)
- [CMDAR: A Chinese Multi-scene Dynamic Audio Reasoning Benchmark with Diverse Challenges](https://mlsanity.com/paper/2509.22461/) (2025)
- [ORCA: Open-ended Response Correctness Assessment for Audio Question Answering](https://mlsanity.com/paper/2512.09066/) (2025)
- [Reflecting Twice before Speaking with Empathy: Self-Reflective Alternating Inference for Empathy-Aware End-to-End Spoken Dialogue](https://mlsanity.com/paper/2601.18281/) (2026)
- [MUGEN: Evaluating and Improving Multi-audio Understanding of Large Audio-Language Models](https://mlsanity.com/paper/2603.09714/) (2026)
- [Nudging Hidden States: Training-Free Model Steering for Chain-of-Thought Reasoning in Large Audio-Language Models](https://mlsanity.com/paper/2603.14636/) (2026)
- [How Auditory Knowledge in LLM Backbones Shapes Audio Language Models: A Holistic Evaluation](https://mlsanity.com/paper/2603.19195/) (2026)
- [Membership Inference Attacks against Large Audio Language Models](https://mlsanity.com/paper/2603.28378/) (2026)
- [MedMosaic: A Challenging Large Scale Benchmark of Diverse Medical Audio](https://mlsanity.com/paper/2605.00969/) (2026)

## Abstract (arXiv)

We introduce DeSTA2.5-Audio, a general-purpose Large Audio Language Model (LALM) designed for robust auditory perception and instruction-following. Recent LALMs augment Large Language Models (LLMs) with auditory capabilities by training on large-scale audio-instruction datasets. However, existing LALMs have often suffered from the catastrophic forgetting of the LLM's original abilities. Therefore, balancing knowledge retention and audio perception has become a critical challenge. To address this, we revisit the data construction pipeline and propose a self-generated cross-modal alignment strategy in which the backbone LLM generates its own training targets, named DeSTA. This approach aims at preserving the LLM's native language proficiency thereby enabling zero-shot generalization without task-specific tuning. We construct DeSTA-AQA5M, a large-scale, task-agnostic dataset containing 5 million training samples derived from 7,000 hours of audio spanning 50 diverse datasets, including speech, environmental sounds, and music. DeSTA2.5-Audio achieves state-of-the-art or competitive performance across a wide range of audio-language benchmarks, including Dynamic-SUPERB, MMAU, SAKURA, Speech-IFEval, and VoiceBench. Comprehensive comparative studies demonstrate that our self-generated strategy outperforms existing training strategies. Our findings underscore the importance of carefully designed data construction in LALM development and offer practical insights for building robust, general-purpose LALMs.