mlsanityData to 7 Oct 2026

AVCap: Leveraging Audio-Visual Features as Text Tokens for Captioning

arXiv 2407.07801 ↗First posted 10 Jul 2024eess.AS cs.CL cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

In recent years, advancements in representation learning and language models have propelled Automated Captioning (AC) to new heights, enabling the generation of human-level descriptions. Leveraging these advancements, we propose AVCap, an Audio-Visual Captioning framework, a simple yet powerful baseline approach applicable to audio-visual captioning. AVCap utilizes audio-visual features as text tokens, which has many advantages not only in performance but also in the extensibility and scalability of the model. AVCap is designed around three pivotal dimensions: the exploration of optimal audio-visual encoder architectures, the adaptation of pre-trained models according to the characteristics of generated text, and the investigation into the efficacy of modality fusion in captioning. Our method outperforms existing audio-visual captioning methods across all metrics and the code is available on https://github.com/JongSuk1/AVCap

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 21 of its 27 references that are in the map. First posted on arXiv 10 Jul 2024.

Builds on

21 of 27 references.

References · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  5. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  6. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  7. 2022Contrastive Audio-Visual Masked Autoencodercited by 142
  8. 2020Multi-modal Dense Video Captioningcited by 113
  9. 2022MAViL: Masked Audio-Video Learnerscited by 56
  10. 2021Audio Captioning Transformercited by 53
  11. 2022Audiovisual Masked Autoencoderscited by 47
  12. 2023Prefix tuning for automated audio captioningcited by 26
9 more
  1. 2020A Transformer-based Audio Captioning Model with Keyword Estimationcited by 24
  2. 2019Audio Caption: Listen and Tellcited by 20
  3. 2021CL4AC: A Contrastive Loss for Audio Captioningcited by 16
  4. 2022Interactive Audio-text Representation for Automated Audio Captioning with Contrastive Learningcited by 15
  5. 2021Automated Audio Captioning using Transfer Learning and Reconstruction Latent Space Similarity Regularizationcited by 12
  6. 2022Visually-Aware Audio Captioning With Adaptive Audio-Visual Attentioncited by 11
  7. 2022Local Information Assisted Attention-free Decoder for Audio Captioningcited by 5
  8. 2023Dual Transformer Decoder based Features Fusion Network for Automated Audio Captioningcited by 3
  9. 2022Automated Audio Captioning via Fusion of Low- and High- Dimensional Featurescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 1.

6 papers.

  1. 2025Mettle: Meta-Token Learning for Memory-Efficient Audio-Visual Adaptationcited by 10
  2. 2025LAVCap: LLM-based Audio-Visual Captioning using Optimal Transportcited by 7
  3. 2025Mitigating Audiovisual Mismatch in Visual-Guide Audio Captioningcited by 3
  4. 2025Hearing from Silence: Reasoning Audio Descriptions from Silent Videos via Vision-Language Modelcited by 1
  5. 2026LAMB: LLM-based Audio Captioning with Modality Gap Bridging via Cauchy-Schwarz Divergencecited by 1
  6. 2025Listening without Looking: Modality Bias in Audio-Visual Captioningcited by 0