# UniAudio 1.5: Large Language Model-driven Audio Codec is A Few-shot Audio Task Learner

arXiv: https://arxiv.org/abs/2406.10056 · HTML: https://mlsanity.com/paper/2406.10056/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Jun 2024.

## Built on by (most cited first, 22 of 22)

- [Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systems](https://mlsanity.com/paper/2504.01990/) (2025)
- [WavChat: A Survey of Spoken Dialogue Models](https://mlsanity.com/paper/2411.13577/) (2024)
- [Recent Advances in Discrete Speech Tokens: A Review](https://mlsanity.com/paper/2502.06490/) (2025)
- [Audio Large Language Models Can Be Descriptive Speech Quality Evaluators](https://mlsanity.com/paper/2501.17202/) (2025)
- [XY-Tokenizer: Mitigating the Semantic-Acoustic Conflict in Low-Bitrate Speech Codecs](https://mlsanity.com/paper/2506.23325/) (2025)
- [UniPose: A Unified Multimodal Framework for Human Pose Comprehension, Generation and Editing](https://mlsanity.com/paper/2411.16781/) (2024)
- [Slamming: Training a Speech Language Model on One GPU in a Day](https://mlsanity.com/paper/2502.15814/) (2025)
- [Do we really have to filter out random noise in pre-training data for language models?](https://mlsanity.com/paper/2502.06604/) (2025)
- [Accelerating Codec-based Speech Synthesis with Multi-Token Prediction and Speculative Decoding](https://mlsanity.com/paper/2410.13839/) (2024)
- [ESPnet-SpeechLM: An Open Speech Language Model Toolkit](https://mlsanity.com/paper/2502.15218/) (2025)
- [LSCodec: Low-Bitrate and Speaker-Decoupled Discrete Speech Codec](https://mlsanity.com/paper/2410.15764/) (2024)
- [LLMs are One-Shot URL Classifiers and Explainers](https://mlsanity.com/paper/2409.14306/) (2024)
- [Robust and Efficient Autoregressive Speech Synthesis with Dynamic Chunk-wise Prediction Policy](https://mlsanity.com/paper/2506.22023/) (2025)
- [StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMs](https://mlsanity.com/paper/2509.22220/) (2025)
- [Towards LLM-Centric Multimodal Fusion: A Survey on Integration Strategies and Techniques](https://mlsanity.com/paper/2506.04788/) (2025)
- [U-Codec: Ultra Low Frame-rate Neural Speech Codec for Fast High-fidelity Speech Generation](https://mlsanity.com/paper/2510.16718/) (2025)
- [MetaSICL: Globalizing Auditory LLMs for Underserved Speakers and Languages via Meta Speech In-Context Learning](https://mlsanity.com/paper/2601.18904/) (2026)
- [AI-Invented Tonal Languages: Preventing a Machine Lingua Franca Beyond Human Understanding](https://mlsanity.com/paper/2503.01063/) (2025)
- [CALM: Class-Conditional Sparse Attention Vectors for Large Audio-Language Models](https://mlsanity.com/paper/2602.07077/) (2026)
- [ATIR: Towards Audio-Text Interleaved Contextual Retrieval](https://mlsanity.com/paper/2604.20267/) (2026)
- [Noisy-Channel Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2607.05198/) (2026)
- [TokenMapper: A Step Toward Interoperable Speech Token Translation](https://mlsanity.com/paper/2609.12563/) (2026)

## Abstract (arXiv)

The Large Language models (LLMs) have demonstrated supreme capabilities in text understanding and generation, but cannot be directly applied to cross-modal tasks without fine-tuning. This paper proposes a cross-modal in-context learning approach, empowering the frozen LLMs to achieve multiple audio tasks in a few-shot style without any parameter update. Specifically, we propose a novel and LLMs-driven audio codec model, LLM-Codec, to transfer the audio modality into the textual space, textiti.e. representing audio tokens with words or sub-words in the vocabulary of LLMs, while keeping high audio reconstruction quality. The key idea is to reduce the modality heterogeneity between text and audio by compressing the audio modality into a well-trained LLMs token space. Thus, the audio representation can be viewed as a new textitforeign language, and LLMs can learn the new textitforeign language with several demonstrations. In experiments, we investigate the performance of the proposed approach across multiple audio understanding and generation tasks, textite.g. speech emotion classification, audio classification, text-to-speech generation, speech enhancement, etc. The experimental results demonstrate that the LLMs equipped with the proposed LLM-Codec, named as UniAudio 1.5, prompted by only a few examples, can achieve the expected functions in simple scenarios. It validates the feasibility and effectiveness of the proposed cross-modal in-context learning approach. To facilitate research on few-shot audio task learning and multi-modal LLMs, we have open-sourced the LLM-Codec model.