mlsanityData to 14 Oct 2026

CARD: Cross-component Audio Representation Distillation for Encoder-Free Audio Captioning

arXiv 2607.04619 ↗First posted 6 Jul 2026cs.SD cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Modern automated audio captioning systems pair a frozen audio encoder with a large language model (LLM) via a trainable projector, incurring the encoder's inference cost and bottlenecking the model through its fixed acoustic features. We present CARD, an encoder-free audio captioning model that removes the encoder at inference: a 13.2M projector feeds a frozen LLM with merged LoRA adapters, while the teacher used to train it is discarded. CARD distills a pre-trained audio teacher (CLAP-HTSAT) into the model, but rather than injecting it into the LLM alone, it routes the teacher's representations across components: perceptual stages to the projector and semantic stages to the LLM. This placement improves CIDEr-D by +11.9 over an LLM-only distilled model on AudioCaps and by +5.0 on Clotho, reaching 55.4 against a 66.4 encoder-kept upper bound with no encoder at inference, showing that where a teacher's knowledge is placed matters as much as its presence.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 31 references that are in the map. First posted on arXiv 6 Jul 2026.

Builds on

25 of 31 references.

References · 25

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  6. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  7. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  8. 2019Clotho: An Audio Captioning Datasetcited by 358
  9. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 267
  10. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 176
  11. 2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 139
  12. 2016Improved Image Captioning via Policy Gradient optimization of SPIDErcited by 113
13 more
  1. 2024Unveiling Encoder-Free Vision-Language Modelscited by 94
  2. 2022Clotho-AQA: A Crowdsourced Dataset for Audio Question Answeringcited by 85
  3. 2024Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-trainingcited by 82
  4. 2025Infinity Instruct: Scaling Instruction Selection and Synthesis to Enhance Language Modelscited by 50
  5. 2025Vision as LoRAcited by 37
  6. 2023Auto-ACD: A Large-scale Dataset for Audio-Language Representation Learningcited by 33
  7. 2023CED: Consistent ensemble distillation for audio taggingcited by 29
  8. 2024EnCLAP: Combining Neural Audio Codec and Audio-Text Joint Embedding for Automated Audio Captioningcited by 28
  9. 2024SLAM-AAC: Enhancing Audio Captioning with Paraphrasing Augmentation and CLAP-Refine through LLMscited by 15
  10. 2024Enhancing Automated Audio Captioning via Large Language Models with Optimized Audio Encodingcited by 14
  11. 2024DRCap: Decoding CLAP Latents with Retrieval-Augmented Generation for Zero-shot Audio Captioningcited by 11
  12. 2026LLM can Read Spectrogram: Encoder-free Speech-Language Modelingcited by 1
  13. 2026AuRA: Internalizing Audio Understanding into LLMs as LoRAcited by 1

Built on by

Not cited yet.