# Mizar: A 159M-Parameter Audio-Language Model for Audio Understanding

arXiv: https://arxiv.org/abs/2609.28344 · HTML: https://mlsanity.com/paper/2609.28344/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 26 references that are in the map.
- First posted on arXiv 23 Sep 2026.

## Builds on (most cited first, 21 of 21)

- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [Large Language Models Are Not Robust Multiple Choice Selectors](https://mlsanity.com/paper/2309.03882/) (2023)
- [BEATs: Audio Pre-Training with Acoustic Tokenizers](https://mlsanity.com/paper/2212.09058/) (2022)
- [SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model](https://mlsanity.com/paper/2502.02737/) (2025)
- [MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2410.19168/) (2024)
- [Listen, Think, and Understand](https://mlsanity.com/paper/2305.10790/) (2023)
- [Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities](https://mlsanity.com/paper/2402.01831/) (2024)
- [HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection](https://mlsanity.com/paper/2202.00874/) (2022)
- [Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities](https://mlsanity.com/paper/2503.03983/) (2025)
- [MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix](https://mlsanity.com/paper/2505.13032/) (2025)
- [Reinforcement Learning Outperforms Supervised Fine-Tuning: A Case Study on Audio Question Answering](https://mlsanity.com/paper/2503.11197/) (2025)
- [Omni-R1: Do You Really Need Audio to Fine-Tune Your Audio LLM?](https://mlsanity.com/paper/2505.09439/) (2025)
- [MELTing point: Mobile Evaluation of Language Transformers](https://mlsanity.com/paper/2403.12844/) (2024)
- [SARI: Structured Audio Reasoning via Curriculum-Guided Reinforcement Learning](https://mlsanity.com/paper/2504.15900/) (2025)
- [CED: Consistent ensemble distillation for audio tagging](https://mlsanity.com/paper/2308.11957/) (2023)
- [Mellow: a small audio language model for reasoning](https://mlsanity.com/paper/2503.08540/) (2025)
- [M2UGen: Multi-modal Music Understanding and Generation with the Power of Large Language Models](https://mlsanity.com/paper/2311.11255/) (2023)
- [Paralinguistic Privacy Protection at the Edge](https://mlsanity.com/paper/2011.02930/) (2020)
- [Summary of DCASE 2026 Task 5: Audio-Dependent Question Answering](https://mlsanity.com/paper/2607.18718/) (2026)

## Abstract (arXiv)

Audio-language models (ALMs) integrate acoustic perception with the knowledge encoded in language models, enabling contextual understanding of auditory events. Making these capabilities practical on devices with limited memory and computation motivates our focus on small ALMs with fewer than 200M parameters. We introduce a recipe that brings together architecture, data, and three-stage training to build Mizar, a 159.3M-parameter ALM. Its architecture connects a compact CED-Small audio encoder to SmolLM2-135M through a frequency-merging mapper. With supervision drawn from ReasonAQA, AudioMCQ, and AVQA, the model undergoes three training stages: audio-language alignment (Stage 1), audio-dependent fine-tuning (Stage 2), and post-training (Stage 3) aimed at strengthening weak skills while retaining learned capabilities. Across five random seeds, Mizar achieves mean accuracies of 52.92% on MMAU, 42.42% on MMAR, and 36.02% on ADQA-clean, surpassing the previous best-performing ALM below 200M parameters on all three benchmarks. It also supports local inference on a single CPU: on questions from the MMAU benchmark, the mean latency from opening the audio file to generating a complete answer is 1.09 seconds. Code and checkpoints are available at https://github.com/KaiyangLi1992/Mizar_159M.