mlsanityData to 7 Oct 2026

Video ChatCaptioner: Towards Enriched Spatiotemporal Descriptions

arXiv 2304.04227 ↗First posted 9 Apr 2023cs.CV cs.AIcited by 42

Not on the map. Why?

Abstract (arXiv)

Video captioning aims to convey dynamic scenes from videos using natural language, facilitating the understanding of spatiotemporal information within our environment. Although there have been recent advances, generating detailed and enriched video descriptions continues to be a substantial challenge. In this work, we introduce Video ChatCaptioner, an innovative approach for creating more comprehensive spatiotemporal video descriptions. Our method employs a ChatGPT model as a controller, specifically designed to select frames for posing video content-driven questions. Subsequently, a robust algorithm is utilized to answer these visual queries. This question-answer framework effectively uncovers intricate video details and shows promise as a method for enhancing video content. Following multiple conversational rounds, ChatGPT can summarize enriched video content based on previous conversations. We qualitatively demonstrate that our Video ChatCaptioner can generate captions containing more visual details about the videos. The code is publicly available at https://github.com/Vision-CAIR/ChatCaptioner

Key facts. Cited by 42 papers in the map (data through 7 Oct 2026). Builds on 36 of its 44 references that are in the map. First posted on arXiv 9 Apr 2023.

Builds on

36 of 44 references.

References · 36

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  7. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  8. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  9. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  10. 2021Florence: A New Foundation Model for Computer Visioncited by 802
  11. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  12. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
24 more
  1. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  2. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  3. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
  4. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  5. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 511
  6. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  7. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  8. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  9. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  10. 2022Visual Language Maps for Robot Navigationcited by 362
  11. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  12. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  13. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  14. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  15. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  16. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  17. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  18. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  19. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  20. 2022End-to-end Generative Pretraining for Multimodal Video Captioningcited by 115
  21. 2023ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptionscited by 86
  22. 2022Bridging Video-text Retrieval with Multiple Choice Questionscited by 79
  23. 2023DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agentscited by 52
  24. 2018SVIn2: An Underwater SLAM System using Sonar, Visual, Inertial, and Depth Sensorcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 22; 2025: 6; 2026: 3.

42 papers.

Sort:

By area:

  1. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  2. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  3. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  4. 2024VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videoscited by 242
  5. 2023Video Understanding with Large Language Models: A Surveycited by 203
  6. 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 176
  7. 2024LongVLM: Efficient Long Video Understanding via Large Language Modelscited by 146
  8. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  9. 2024MotionLLM: Understanding Human Behaviors from Human Motions and Videoscited by 89
  10. 2023ChatVideo: A Tracklet-centric Multimodal and Versatile Video Understanding Systemcited by 65
  11. 2024DrVideo: Document Retrieval Based Long Video Understandingcited by 63
  12. 2024OpenBias: Open-set Bias Detection in Text-to-Image Generative Modelscited by 61
30 more
  1. 2023Exploring Human-Like Translation Strategy with Large Language Modelscited by 44
  2. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
  3. 2023LLM as A Robotic Brain: Unifying Egocentric Memory and Controlcited by 32
  4. 2024Referring Atomic Video Action Recognitioncited by 16
  5. 2024VLLMs Provide Better Context for Emotion Understanding Through Common Sense Reasoningcited by 15
  6. 2024Contextual AD Narration with Interleaved Multimodal Sequencecited by 14
  7. 2024MMTrail: A Multimodal Trailer Video Dataset with Language and Music Descriptionscited by 13
  8. 2024HOI-Ref: Hand-Object Interaction Referral in Egocentric Visioncited by 9
  9. 2025Video-LMM Post-Training: A Deep Dive into Video Reasoning with Large Multimodal Modelscited by 8
  10. 2024LLAVIDAL: A Large LAnguage VIsion Model for Daily Activities of Livingcited by 7
  11. 2025Fine-Grained Captioning of Long Videos through Scene Graph Consolidationcited by 7
  12. 2024Prompting Large Vision-Language Models for Compositional Reasoningcited by 6
  13. 2023Affective Visual Dialog: A Large-Scale Benchmark for Emotional Reasoning Based on Visually Grounded Conversationscited by 4
  14. 2024From Linguistic Giants to Sensory Maestros: A Survey on Cross-Modal Reasoning with Large Language Modelscited by 4
  15. 2024Agent-based Video Trimmingcited by 4
  16. 2025PhyVLLM: Physics-Guided Video Language Model with Motion-Appearance Disentanglementcited by 4
  17. 2026Soap2Soap: Long Cinematic Video Remaking via Multi-Agent Collaborationcited by 4
  18. 2023Language as the Medium: Multimodal Video Classification through text onlycited by 3
  19. 2024UAL-Bench: The First Comprehensive Unusual Activity Localization Benchmarkcited by 3
  20. 2023Enhance Reasoning Ability of Visual-Language Models via Large Language Modelscited by 2
  21. 2024IWISDM: Assessing instruction following in multimodal models at scalecited by 2
  22. 2024GradBias: Unveiling Word Influence on Bias in Text-to-Image Generative Modelscited by 2
  23. 2024Character-aware audio-visual subtitling in contextcited by 2
  24. 2025NoteIt: A System Converting Instructional Videos to Interactable Notes Through Multimodal Video Understandingcited by 2
  25. 2026Towards Sparse Video Understanding and Reasoningcited by 2
  26. 2024G2TR: Generalized Grounded Temporal Reasoning for Robot Instruction Following by Combining Large Pre-trained Modelscited by 1
  27. 2024Training-Free Action Recognition and Goal Inference with Dynamic Frame Selectioncited by 0
  28. 2025ImageSet2Text: Describing Sets of Images through Textcited by 0
  29. 2025ViMoNet: A Multimodal Vision-Language Framework for Human Behavior Understanding from Motion and Videocited by 0
  30. 2026TimeProVe: Propose, then Verify for Efficient Long Video Temporal Reasoning in Activities of Daily Livingcited by 0