# Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning

arXiv: https://arxiv.org/abs/2003.05162 · HTML: https://mlsanity.com/paper/2003.05162/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Mar 2020.

## Built on by (most cited first, 30 of 36)

- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation](https://mlsanity.com/paper/2109.06835/) (2021)
- [Compressing Visual-linguistic Model via Knowledge Distillation](https://mlsanity.com/paper/2104.02096/) (2021)
- [Video Question Answering: Datasets, Algorithms and Challenges](https://mlsanity.com/paper/2203.01225/) (2022)
- [Multimodal learning with graphs](https://mlsanity.com/paper/2209.03299/) (2022)
- [From Representation to Reasoning: Towards both Evidence and Commonsense Reasoning for Video Question-Answering](https://mlsanity.com/paper/2205.14895/) (2022)
- [Injecting Semantic Concepts into End-to-End Image Captioning](https://mlsanity.com/paper/2112.05230/) (2021)
- [Benchmarks for Automated Commonsense Reasoning: A Survey](https://mlsanity.com/paper/2302.04752/) (2023)
- [The Abduction of Sherlock Holmes: A Dataset for Visual Abductive Reasoning](https://mlsanity.com/paper/2202.04800/) (2022)
- [A Review on Methods and Applications in Multimodal Deep Learning](https://mlsanity.com/paper/2202.09195/) (2022)
- [GOAL: A Challenging Knowledge-grounded Video Captioning Benchmark for Real-time Soccer Commentary Generation](https://mlsanity.com/paper/2303.14655/) (2023)
- [Recent Advances and Trends in Multimodal Deep Learning: A Review](https://mlsanity.com/paper/2105.11087/) (2021)
- [End-to-end Knowledge Retrieval with Multi-modal Queries](https://mlsanity.com/paper/2306.00424/) (2023)
- [WinoGAViL: Gamified Association Benchmark to Challenge Vision-and-Language Models](https://mlsanity.com/paper/2207.12576/) (2022)
- [Fine-grained Audible Video Description](https://mlsanity.com/paper/2303.15616/) (2023)
- [CLEVR_HYP: A Challenge Dataset and Baselines for Visual Question Answering with Hypothetical Actions over Images](https://mlsanity.com/paper/2104.05981/) (2021)
- [CRIPP-VQA: Counterfactual Reasoning about Implicit Physical Properties via Video Question Answering](https://mlsanity.com/paper/2211.03779/) (2022)
- [NarrativeBridge: Enhancing Video Captioning with Causal-Temporal Narrative](https://mlsanity.com/paper/2406.06499/) (2024)
- [TOGA: Temporally Grounded Open-Ended Video QA with Weak Supervision](https://mlsanity.com/paper/2506.09445/) (2025)
- [WhyAct: Identifying Action Reasons in Lifestyle Vlogs](https://mlsanity.com/paper/2109.02747/) (2021)
- [Audio Description Generation in the Era of LLMs and VLMs: A Review of Transferable Generative AI Technologies](https://mlsanity.com/paper/2410.08860/) (2024)
- [iReason: Multimodal Commonsense Reasoning using Videos and Natural Language with Interpretability](https://mlsanity.com/paper/2107.10300/) (2021)
- [Hybrid Reasoning Network for Video-based Commonsense Captioning](https://mlsanity.com/paper/2108.02365/) (2021)
- [Commonsense for Zero-Shot Natural Language Video Localization](https://mlsanity.com/paper/2312.17429/) (2023)
- [Looking Beyond Visible Cues: Implicit Video Question Answering via Dual-Clue Reasoning](https://mlsanity.com/paper/2506.07811/) (2025)
- [Unsupervised Pronoun Resolution via Masked Noun-Phrase Prediction](https://mlsanity.com/paper/2105.12392/) (2021)
- [Materialized Knowledge Bases from Commonsense Transformers](https://mlsanity.com/paper/2112.14815/) (2021)
- [To Find Waldo You Need Contextual Cues: Debiasing Who's Waldo](https://mlsanity.com/paper/2203.16682/) (2022)
- [Knowledge Guided Entity-aware Video Captioning and A Basketball Benchmark](https://mlsanity.com/paper/2401.13888/) (2024)

## Abstract (arXiv)

Captioning is a crucial and challenging task for video understanding. In videos that involve active agents such as humans, the agent's actions can bring about myriad changes in the scene. Observable changes such as movements, manipulations, and transformations of the objects in the scene, are reflected in conventional video captioning. Unlike images, actions in videos are also inherently linked to social aspects such as intentions (why the action is taking place), effects (what changes due to the action), and attributes that describe the agent. Thus for video understanding, such as when captioning videos or when answering questions about videos, one must have an understanding of these commonsense aspects. We present the first work on generating commonsense captions directly from videos, to describe latent aspects such as intentions, effects, and attributes. We present a new dataset "Video-to-Commonsense (V2C)" that contains ~9k videos of human agents performing various actions, annotated with 3 types of commonsense descriptions. Additionally we explore the use of open-ended video-based commonsense question answering (V2C-QA) as a way to enrich our captions. Both the generation task and the QA task can be used to enrich video captions.