# LLMVA-GEBC: Large Language Model with Video Adapter for Generic Event Boundary Captioning

arXiv: https://arxiv.org/abs/2306.10354 · HTML: https://mlsanity.com/paper/2306.10354/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 13 references that are in the map.
- First posted on arXiv 17 Jun 2023.

## Builds on (most cited first, 10 of 10)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [InternVideo: General Video Foundation Models via Generative and Discriminative Learning](https://mlsanity.com/paper/2212.03191/) (2022)
- [Omnivore: A Single Model for Many Visual Modalities](https://mlsanity.com/paper/2201.08377/) (2022)
- [Exploiting Context Information for Generic Event Boundary Captioning](https://mlsanity.com/paper/2207.01050/) (2022)
- [Dual-Stream Transformer for Generic Event Boundary Captioning](https://mlsanity.com/paper/2207.03038/) (2022)

## Built on by (most cited first, 9 of 9)

- [Video Understanding with Large Language Models: A Survey](https://mlsanity.com/paper/2312.17432/) (2023)
- [CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputs](https://mlsanity.com/paper/2401.02582/) (2024)
- [V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuning](https://mlsanity.com/paper/2404.12353/) (2024)
- [ProVision: Programmatically Scaling Vision-centric Instruction Data for Multimodal Language Models](https://mlsanity.com/paper/2412.07012/) (2024)
- [An Empirical Analysis on Large Language Models in Debate Evaluation](https://mlsanity.com/paper/2406.00050/) (2024)
- [Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding](https://mlsanity.com/paper/2403.16276/) (2024)
- [OSCaR: Object State Captioning and State Change Representation](https://mlsanity.com/paper/2402.17128/) (2024)
- [EAGLE: Egocentric AGgregated Language-video Engine](https://mlsanity.com/paper/2409.17523/) (2024)
- [Investigating the Scaling Effect of Instruction Templates for Training Multimodal Language Model](https://mlsanity.com/paper/2412.08307/) (2024)

## Abstract (arXiv)

Our winning entry for the CVPR 2023 Generic Event Boundary Captioning (GEBC) competition is detailed in this paper. Unlike conventional video captioning tasks, GEBC demands that the captioning model possess an understanding of immediate changes in status around the designated video boundary, making it a difficult task. This paper proposes an effective model LLMVA-GEBC (Large Language Model with Video Adapter for Generic Event Boundary Captioning): (1) We utilize a pretrained LLM for generating human-like captions with high quality. (2) To adapt the model to the GEBC task, we take the video Q-former as an adapter and train it with the frozen visual feature extractors and LLM. Our proposed method achieved a 76.14 score on the test set and won the first place in the challenge. Our code is available at https://github.com/zjr2000/LLMVA-GEBC .