# AQAScore: Evaluating Semantic Alignment in Text-to-Audio Generation via Audio Question Answering

arXiv: https://arxiv.org/abs/2601.14728 · HTML: https://mlsanity.com/paper/2601.14728/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 39 references that are in the map.
- First posted on arXiv 21 Jan 2026.

## Builds on (most cited first, 30 of 34)

- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs](https://mlsanity.com/paper/2503.01743/) (2025)
- [Evaluating Text-to-Visual Generation with Image-to-Text Generation](https://mlsanity.com/paper/2404.01291/) (2024)
- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2410.19168/) (2024)
- [Diffsound: Discrete Diffusion Model for Text-to-sound Generation](https://mlsanity.com/paper/2207.09983/) (2022)
- [On The Landscape of Spoken Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.08528/) (2025)
- [Vision-Language Models Do Not Understand Negation](https://mlsanity.com/paper/2501.09425/) (2025)
- [Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks](https://mlsanity.com/paper/2411.05361/) (2024)
- [TangoFlux: Super Fast and Faithful Text to Audio Generation with Flow Matching and Clap-Ranked Preference Optimization](https://mlsanity.com/paper/2412.21037/) (2024)
- [AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs](https://mlsanity.com/paper/2311.06753/) (2023)
- [CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models](https://mlsanity.com/paper/2310.08753/) (2023)
- [Can Audio Captions Be Evaluated with Image Caption Metrics?](https://mlsanity.com/paper/2110.04684/) (2021)
- [Understanding Sounds, Missing the Questions: The Challenge of Object Hallucination in Large Audio-Language Models](https://mlsanity.com/paper/2406.08402/) (2024)
- [Can Large Audio-Language Models Truly Hear? Tackling Hallucinations with Multi-Task Assessment and Stepwise Audio Reasoning](https://mlsanity.com/paper/2410.16130/) (2024)
- [Speech-Copilot: Leveraging Large Language Models for Speech Processing via Task Decomposition, Modularization, and Program Generation](https://mlsanity.com/paper/2407.09886/) (2024)
- [Audio Large Language Models Can Be Descriptive Speech Quality Evaluators](https://mlsanity.com/paper/2501.17202/) (2025)
- [The Hard Positive Truth about Vision-Language Compositionality](https://mlsanity.com/paper/2409.17958/) (2024)
- [AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluation](https://mlsanity.com/paper/2507.12705/) (2025)
- [PAM: Prompting Audio-Language Models for Audio Quality Assessment](https://mlsanity.com/paper/2402.00282/) (2024)
- [MusicEval: A Generative Music Dataset with Expert Ratings for Automatic Text-to-Music Evaluation](https://mlsanity.com/paper/2501.10811/) (2025)
- [ETTA: Elucidating the Design Space of Text-to-Audio Models](https://mlsanity.com/paper/2412.19351/) (2024)
- [Audio-Aware Large Language Models as Judges for Speaking Styles](https://mlsanity.com/paper/2506.05984/) (2025)
- [Game-Time: Evaluating Temporal Dynamics in Spoken Language Models](https://mlsanity.com/paper/2509.26388/) (2025)
- [Teaching Audio-Aware Large Language Models What Does Not Hear: Mitigating Hallucinations through Synthesized Negative Samples](https://mlsanity.com/paper/2505.14518/) (2025)
- [From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Data](https://mlsanity.com/paper/2505.20166/) (2025)
- [TTA-Bench: A Comprehensive Benchmark for Evaluating Text-to-Audio Models](https://mlsanity.com/paper/2509.02398/) (2025)
- [BRACE: A Benchmark for Robust Audio Caption Quality Evaluation](https://mlsanity.com/paper/2512.10403/) (2025)
- [Towards General-Purpose Text-Instruction-Guided Voice Conversion](https://mlsanity.com/paper/2309.14324/) (2023)

## Built on by (most cited first, 8 of 8)

- [Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Models](https://mlsanity.com/paper/2604.25591/) (2026)
- [AnyAudio-Judge: A Dynamic Rubric-Based Benchmark and Evaluator for Audio Instruction Following](https://mlsanity.com/paper/2606.03116/) (2026)
- [The Sound of Absence: Audio-Language Embedding Models Struggle with Negation](https://mlsanity.com/paper/2607.12290/) (2026)
- [CAF-Score: Calibrating CLAP with LALMs for Reference-free Audio Captioning Evaluation](https://mlsanity.com/paper/2603.19615/) (2026)
- [Improving Text-to-Audio Instruction Following via Fine-Grained Feedback from Audio-Aware Large Language Models](https://mlsanity.com/paper/2607.13408/) (2026)
- [TORUS: A Test of Rendering-Understanding Self-Coherence for Unified Audio Models](https://mlsanity.com/paper/2607.28896/) (2026)
- [AudioScape-TTA: A Structured Soundscape Benchmark for Fine-Grained Text-to-Audio Evaluation](https://mlsanity.com/paper/2608.04479/) (2026)
- [MIRA: A Musical Intent Refinement Agent for Aligning Text-to-Music Generation with User Intent](https://mlsanity.com/paper/2610.10355/) (2026)

## Abstract (arXiv)

Although text-to-audio generation has made remarkable progress in realism and diversity, the development of evaluation metrics has not kept pace. Widely-adopted approaches, typically based on embedding similarity like CLAPScore, effectively measure general relevance but remain limited in fine-grained semantic alignment and compositional reasoning. To address this, we introduce AQAScore, a backbone-agnostic evaluation framework that leverages the reasoning capabilities of audio-aware large language models (ALLMs). AQAScore reformulates assessment as a probabilistic semantic verification task; rather than relying on open-ended text generation, it estimates alignment by computing the exact log-probability of a "Yes" answer to targeted semantic queries. We evaluate AQAScore across multiple benchmarks, including human-rated relevance, pairwise comparison, and compositional reasoning tasks. Experimental results show that AQAScore consistently achieves higher correlation with human judgments than similarity-based metrics and generative prompting baselines, showing its effectiveness in capturing subtle semantic inconsistencies and scaling with the capability of underlying ALLMs.