# When Noise Lowers The Loss: Rethinking Likelihood-Based Evaluation in Music Large Language Models

arXiv: https://arxiv.org/abs/2602.02738 · HTML: https://mlsanity.com/paper/2602.02738/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 22 references that are in the map.
- First posted on arXiv 2 Feb 2026.

## Builds on (most cited first, 20 of 20)

- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [GPTScore: Evaluate as You Desire](https://mlsanity.com/paper/2302.04166/) (2023)
- [Meta Audiobox Aesthetics: Unified Automatic Quality Assessment for Speech, Music, and Sound](https://mlsanity.com/paper/2502.05139/) (2025)
- [Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generation](https://mlsanity.com/paper/2204.01171/) (2022)
- [The Generative AI Paradox: "What It Can Create, It May Not Understand"](https://mlsanity.com/paper/2311.00059/) (2023)
- [Adapting Frechet Audio Distance for Generative Music Evaluation](https://mlsanity.com/paper/2311.01616/) (2023)
- [YuE: Scaling Open Foundation Models for Long-Form Music Generation](https://mlsanity.com/paper/2503.08638/) (2025)
- [Do LLMs Overcome Shortcut Learning? An Evaluation of Shortcut Challenges in Large Language Models](https://mlsanity.com/paper/2410.13343/) (2024)
- [Aligning Text-to-Music Evaluation with Human Preferences](https://mlsanity.com/paper/2503.16669/) (2025)
- [CMI-Bench: A Comprehensive Benchmark for Evaluating Music Instruction Following](https://mlsanity.com/paper/2506.12285/) (2025)
- [Benchmarking Music Generation Models and Metrics via Human Preference Studies](https://mlsanity.com/paper/2506.19085/) (2025)
- [Survey on the Evaluation of Generative Models in Music](https://mlsanity.com/paper/2506.05104/) (2025)
- [Likelihood-based Mitigation of Evaluation Bias in Large Language Models](https://mlsanity.com/paper/2402.15987/) (2024)
- [Music Arena: Live Evaluation for Text-to-Music](https://mlsanity.com/paper/2507.20900/) (2025)
- [Understanding the Role of Cross-Entropy Loss in Fairly Evaluating Large Language Model-based Recommendation](https://mlsanity.com/paper/2402.06216/) (2024)

## Abstract (arXiv)

The rise of music large language models (LLMs) demands robust methods of evaluating output quality, especially in distinguishing high-quality compositions from "garbage music". Curiously, we observe that the standard cross-entropy loss -- a core training metric -- often decrease when models encounter systematically corrupted music, undermining its validity as a standalone quality indicator. To investigate this paradox, we introduce noise injection experiment, where controlled noise signal of varying lengths are injected into musical contexts. We hypothesize that a model's loss reacting positively to these perturbations, specifically a sharp increase ("Peak" area) for short injection, can serve as a proxy for its ability to discern musical integrity. Experiments with MusicGen models in the audio waveform domain confirm that Music LLMs respond more strongly to local, texture-level disruptions than to global semantic corruption. Beyond exposing this bias, our results highlight a new principle: the shape of the loss curve -- rather than its absolute value -- encodes critical information about the quality of the generated content (i.e., model behavior). We envision this profile-based evaluation as a label-free, model-intrinsic framework for assessing musical quality -- opening the door to more principled training objectives and sharper benchmarks.