mlsanityData to 14 Oct 2026

Zero-Shot Video Captioning with Evolving Pseudo-Tokens

arXiv 2207.11100 ↗First posted 22 Jul 2022cs.CVcited by 25

Not on the map. Why?

Abstract (arXiv)

We introduce a zero-shot video captioning method that employs two frozen networks: the GPT-2 language model and the CLIP image-text matching model. The matching score is used to steer the language model toward generating a sentence that has a high average matching score to a subset of the video frames. Unlike zero-shot image captioning methods, our work considers the entire sentence at once. This is achieved by optimizing, during the generation process, part of the prompt from scratch, by modifying the representation of all other tokens in the prompt, and by repeating the process iteratively, gradually improving the specificity and comprehensiveness of the generated sentence. Our experiments show that the generated captions are coherent and display a broad range of real-world knowledge. Our code is available at: https://github.com/YoadTew/zero-shot-video-to-text

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 52 of its 212 references that are in the map. First posted on arXiv 22 Jul 2022.

Builds on

52 of 212 references.

References · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  8. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  11. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  12. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
38 more
  1. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,255
  2. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  3. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  4. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  5. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
  6. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  7. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  8. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  9. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
  10. 2021GPT Understands, Toocited by 716
  11. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
  12. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  13. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  14. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  15. 2020Multi-modal Transformer for Video Retrievalcited by 395
  16. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 376
  17. 2017A simple neural network module for relational reasoningcited by 346
  18. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 334
  19. 2017Temporal Relational Reasoning in Videoscited by 324
  20. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  21. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  22. 2018Auto-Encoding Scene Graphs for Image Captioningcited by 255
  23. 2017Style Transfer from Non-Parallel Text by Cross-Alignmentcited by 248
  24. 2018Exploring Visual Relationship for Image Captioningcited by 215
  25. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  26. 2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
  27. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  28. 2022Revealing Single Frame Bias for Video-and-Language Learningcited by 133
  29. 2014Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)cited by 94
  30. 2022Language Models Can See: Plugging Visual Controls in Text Generationcited by 90
  31. 2021CLIP4Caption: CLIP for Video Captioncited by 88
  32. 2015Video Paragraph Captioning Using Hierarchical Recurrent Neural Networkscited by 54
  33. 2019A Simple Baseline for Audio-Visual Scene-Aware Dialogcited by 52
  34. 2021Image-Based CLIP-Guided Essence Transfercited by 39
  35. 2016Deep Learning for Video Classification and Captioningcited by 28
  36. 2017Trainable Greedy Decoding for Neural Machine Translationcited by 23
  37. 2017High-Order Attention Models for Visual Question Answeringcited by 22
  38. 2014Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotationcited by 16

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 12; 2024: 5; 2025: 3; 2026: 3.

25 papers.

Sort:

By area:

  1. 2022Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?cited by 90
  2. 2023DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Trainingcited by 84
  3. 2023AutoSAM: Adapting SAM to Medical Images by Overloading the Prompt Encodercited by 70
  4. 2024MeaCap: Memory-Augmented Zero-shot Image Captioningcited by 35
  5. 2023VLAB: Enhancing Video Language Pre-training by Feature Adapting and Blendingcited by 17
  6. 2023MultiCapCLIP: Auto-Encoding Prompts for Zero-Shot Multilingual Visual Captioningcited by 13
  7. 2023From Association to Generation: Text-only Captioning by Unsupervised Cross-modal Mappingcited by 11
  8. 2023Zero-shot audio captioning with audio-language model guidance and audio context keywordscited by 11
  9. 2024IFCap: Image-like Retrieval and Frequency-based Entity Filtering for Zero-shot Captioningcited by 11
  10. 2023DisCLIP: Open-Vocabulary Referring Expression Generationcited by 8
  11. 2025Fine-Grained Captioning of Long Videos through Scene Graph Consolidationcited by 8
  12. 2024RETTA: Retrieval-Enhanced Test-Time Adaptation for Zero-Shot Video Captioningcited by 5
13 more
  1. 2023ZeroNLG: Aligning and Autoencoding Domains for Zero-Shot Multimodal and Multilingual Natural Language Generationcited by 4
  2. 2023Zero-Shot Dense Video Captioning by Jointly Optimizing Text and Momentcited by 4
  3. 2023ReSeTOX: Re-learning attention weights for toxicity mitigation in machine translationcited by 3
  4. 2023Zero-Shot Audio Captioning via Audibility Guidancecited by 3
  5. 2023Zero-shot Translation of Attention Patterns in VQA Models to Natural Languagecited by 3
  6. 2024SnapCap: Efficient Snapshot Compressive Video Captioningcited by 2
  7. 2022Describing Sets of Images with Textual-PCAcited by 1
  8. 2024How Vision-Language Tasks Benefit from Large Pre-trained Models: A Surveycited by 0
  9. 2025The Devil is in the Distributions: Explicit Modeling of Scene Content is Key in Zero-Shot Video Captioningcited by 0
  10. 2025LogSTOP: Temporal Scores over Prediction Sequences for Matching and Retrievalcited by 0
  11. 2026MobiDiary: Autoregressive Action Captioning with Wearable Devices and Wireless Signalscited by 0
  12. 2026ConTrans: Learning Text-enhanced Local-global Temporal Representations for Zero-shot Temporal Action Localizationcited by 0
  13. 2026Watching Synthetic Videos: Aligning Cross-modal Representations with Visual Synthesis for Zero-shot Video Captioningcited by 0