mlsanityData to 7 Oct 2026

VLCap: Vision-Language with Contrastive Learning for Coherent Video Paragraph Captioning

arXiv 2206.12972 ↗First posted 26 Jun 2022cs.CVcited by 19

Not on the map. Why?

Abstract (arXiv)

In this paper, we leverage the human perceiving process, that involves vision and language interaction, to generate a coherent paragraph description of untrimmed videos. We propose vision-language (VL) features consisting of two modalities, i.e., (i) vision modality to capture global visual content of the entire scene and (ii) language modality to extract scene elements description of both human and non-human objects (e.g. animals, vehicles, etc), visual and non-visual elements (e.g. relations, activities, etc). Furthermore, we propose to train our proposed VLCap under a contrastive learning VL loss. The experiments and ablation studies on ActivityNet Captions and YouCookII datasets show that our VLCap outperforms existing SOTA methods on both accuracy and diversity metrics.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 23 of its 36 references that are in the map. First posted on arXiv 26 Jun 2022.

Builds on

23 of 36 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  5. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  6. 2018SlowFast Networks for Video Recognitioncited by 1,862
  7. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  8. 2019Contrastive Multiview Codingcited by 1,297
  9. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  10. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,067
  11. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  12. 2017Dense-Captioning Events in Videoscited by 789
11 more
  1. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  2. 2016Convolutional Two-Stream Network Fusion for Video Action Recognitioncited by 384
  3. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
  4. 2021End-to-End Dense Video Captioning with Parallel Decodingcited by 133
  5. 2017Towards a Visual Privacy Advisor: Understanding and Predicting Privacy Risks in Imagescited by 101
  6. 2020COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learningcited by 100
  7. 2020MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioningcited by 79
  8. 2018Grounded Video Descriptioncited by 78
  9. 2018Move Forward and Tell: A Progressive Generator of Video Descriptionscited by 35
  10. 2018Adversarial Inference for Multi-Sentence Video Descriptioncited by 31
  11. 2021AEI: Actors-Environment Interaction with Adaptive Attention for Temporal Action Proposals Generationcited by 11

Built on by

Cited by year

Citing papers per year: 2022: 4; 2023: 6; 2024: 8; 2025: 1.

19 papers.

Sort:

By area:

  1. 2022VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captionerscited by 69
  2. 2022CLIP-TSA: CLIP-Assisted Temporal Self-Attention for Weakly-Supervised Video Anomaly Detectioncited by 67
  3. 2025MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentationcited by 34
  4. 2023A Review of Deep Learning for Video Captioningcited by 28
  5. 2022VLTinT: Visual-Linguistic Transformer-in-Transformer for Coherent Video Paragraph Captioningcited by 25
  6. 2023AerialFormer: Multi-resolution Transformer for Aerial Image Segmentationcited by 23
  7. 2024Narrative Action Evaluation with Prompt-Guided Multimodal Interactioncited by 19
  8. 2024SANER: Annotation-free Societal Attribute Neutralizer for Debiasing CLIPcited by 19
  9. 2023Dense Video Captioning: A Survey of Techniques, Datasets and Evaluation Protocolscited by 17
  10. 2024TrafficVLM: A Controllable Visual Language Model for Traffic Video Captioningcited by 17
  11. 2023ZEETAD: Adapting Pretrained Vision-Language Model for Zero-Shot End-to-End Temporal Action Detectioncited by 12
  12. 2023Z-GMOT: Zero-shot Generic Multiple Object Trackingcited by 7
7 more
  1. 2024R2-Bench: Benchmarking the Robustness of Referring Perception Models under Perturbationscited by 6
  2. 2022Contextual Explainable Video Representation: Human Perception-based Understandingcited by 5
  3. 2024Towards Multimodal Video Paragraph Captioning Models Robust to Missing Modalitycited by 3
  4. 2023SolarFormer: Multi-scale Transformer for Solar PV Profilingcited by 1
  5. 2024CarcassFormer: An End-to-end Transformer-based Framework for Simultaneous Localization, Segmentation and Classification of Poultry Carcass Defectcited by 1
  6. 2024S3Former: Self-supervised High-resolution Transformer for Solar PV Profilingcited by 0
  7. 2024GEM-VPC: A dual Graph-Enhanced Multimodal integration for Video Paragraph Captioningcited by 0