# WhisQ: Cross-Modal Representation Learning for Text-to-Music MOS Prediction

arXiv: https://arxiv.org/abs/2506.05899 · HTML: https://mlsanity.com/paper/2506.05899/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 25 references that are in the map.
- First posted on arXiv 6 Jun 2025.

## Builds on (most cited first, 18 of 18)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [Fast Timing-Conditioned Latent Audio Diffusion](https://mlsanity.com/paper/2402.04825/) (2024)
- [MOSNet: Deep Learning based Objective Assessment for Voice Conversion](https://mlsanity.com/paper/1904.08352/) (2019)
- [MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies](https://mlsanity.com/paper/2308.01546/) (2023)
- [MusicMagus: Zero-Shot Text-to-Music Editing via Diffusion Models](https://mlsanity.com/paper/2402.06178/) (2024)
- [CMOT: Cross-modal Mixup via Optimal Transport for Speech Translation](https://mlsanity.com/paper/2305.14635/) (2023)
- [MusicEval: A Generative Music Dataset with Expert Ratings for Automatic Text-to-Music Evaluation](https://mlsanity.com/paper/2501.10811/) (2025)
- [MOSPC: MOS Prediction Based on Pairwise Comparison](https://mlsanity.com/paper/2306.10493/) (2023)
- [LAVCap: LLM-based Audio-Visual Captioning using Optimal Transport](https://mlsanity.com/paper/2501.09291/) (2025)
- [U-SAM: An audio language Model for Unified Speech, Audio, and Music Understanding](https://mlsanity.com/paper/2505.13880/) (2025)
- [Resource-Efficient Fine-Tuning Strategies for Automatic MOS Prediction in Text-to-Speech for Low-Resource Languages](https://mlsanity.com/paper/2305.19396/) (2023)
- [APG-MOS: Auditory Perception Guided-MOS Predictor for Synthetic Speech](https://mlsanity.com/paper/2504.20447/) (2025)

## Built on by (most cited first, 2 of 2)

- [CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction](https://mlsanity.com/paper/2603.00610/) (2026)
- [Local Multimodal Music Alignment from Global Supervision](https://mlsanity.com/paper/2607.10023/) (2026)

## Abstract (arXiv)

Mean Opinion Score (MOS) prediction for text to music systems requires evaluating both overall musical quality and text prompt alignment. This paper introduces WhisQ, a multimodal architecture that addresses this dual-assessment challenge through sequence level co-attention and optimal transport regularization. WhisQ employs the Whisper Base pretrained model for temporal audio encoding and Qwen 3, a 0.6B Small Language Model (SLM), for text encoding, with both maintaining sequence structure for fine grained cross-modal modeling. The architecture features specialized prediction pathways: OMQ is predicted from pooled audio embeddings, while TA leverages bidirectional sequence co-attention between audio and text. Sinkhorn optimal transport loss further enforce semantic alignment in the shared embedding space. On the MusicEval Track-1 dataset, WhisQ achieves substantial improvements over the baseline: 7% improvement in Spearman correlation for OMQ and 14% for TA. Ablation studies reveal that optimal transport regularization provides the largest performance gain (10% SRCC improvement), demonstrating the importance of explicit cross-modal alignment for text-to-music evaluation.