# GL-RG: Global-Local Representation Granularity for Video Captioning

arXiv: https://arxiv.org/abs/2205.10706 · HTML: https://mlsanity.com/paper/2205.10706/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 35 references that are in the map.
- First posted on arXiv 22 May 2022.

## Builds on (most cited first, 25 of 25)

- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016)
- [CIDEr: Consensus-based Image Description Evaluation](https://mlsanity.com/paper/1411.5726/) (2014)
- [Self-critical Sequence Training for Image Captioning](https://mlsanity.com/paper/1612.00563/) (2016)
- [Sequence to Sequence -- Video to Text](https://mlsanity.com/paper/1505.00487/) (2015)
- [Bridging the Gap between Training and Inference for Neural Machine Translation](https://mlsanity.com/paper/1906.02448/) (2019)
- [Spatio-Temporal Graph for Video Captioning with Knowledge Distillation](https://mlsanity.com/paper/2003.13942/) (2020)
- [Object Relational Graph with Teacher-Recommended Learning for Video Captioning](https://mlsanity.com/paper/2002.11566/) (2020)
- [Reconstruction Network for Video Captioning](https://mlsanity.com/paper/1803.11438/) (2018)
- [Describing Videos by Exploiting Temporal Structure](https://mlsanity.com/paper/1502.08029/) (2015)
- [SG-Net: Spatial Granularity Network for One-Stage Video Instance Segmentation](https://mlsanity.com/paper/2103.10284/) (2021)
- [Memory-Attended Recurrent Network for Video Captioning](https://mlsanity.com/paper/1905.03966/) (2019)
- [Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generation](https://mlsanity.com/paper/1706.09799/) (2017)
- [Less Is More: Picking Informative Frames for Video Captioning](https://mlsanity.com/paper/1803.01457/) (2018)
- [Controllable Video Captioning with POS Sequence Guidance Based on Gated Fusion Network](https://mlsanity.com/paper/1908.10072/) (2019)
- [Video Captioning via Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1711.11135/) (2017)
- [Open-book Video Captioning with Retrieve-Copy-Generate Network](https://mlsanity.com/paper/2103.05284/) (2021)
- [Object-aware Aggregation with Bidirectional Temporal Graph for Video Captioning](https://mlsanity.com/paper/1906.04375/) (2019)
- [Video Relationship Reasoning using Gated Spatio-Temporal Energy Graph](https://mlsanity.com/paper/1903.10547/) (2019)
- [ConvNet Architecture Search for Spatiotemporal Feature Learning](https://mlsanity.com/paper/1708.05038/) (2017)
- [Watch, Listen and Tell: Multi-modal Weakly Supervised Dense Event Captioning](https://mlsanity.com/paper/1909.09944/) (2019)
- [Semantic Grouping Network for Video Captioning](https://mlsanity.com/paper/2102.00831/) (2021)
- [Stacked Spatio-Temporal Graph Convolutional Networks for Action Segmentation](https://mlsanity.com/paper/1811.10575/) (2018)
- [O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioning](https://mlsanity.com/paper/2108.02359/) (2021)
- [Watch It Twice: Video Captioning with a Refocused Video Encoder](https://mlsanity.com/paper/1907.12905/) (2019)

## Built on by (most cited first, 16 of 16)

- [Solve the Puzzle of Instance Segmentation in Videos: A Weakly Supervised Framework with Spatio-Temporal Collaboration](https://mlsanity.com/paper/2212.07592/) (2022)
- [FAQ: Feature Aggregated Queries for Transformer-based Video Object Detectors](https://mlsanity.com/paper/2303.08319/) (2023)
- [FineEHR: Refine Clinical Note Representations to Improve Mortality Prediction](https://mlsanity.com/paper/2304.11794/) (2023)
- [CML-MOTS: Collaborative Multi-task Learning for Multi-Object Tracking and Segmentation](https://mlsanity.com/paper/2311.00987/) (2023)
- [GeoSegNet: Point Cloud Semantic Segmentation via Geometric Encoder-Decoder Modeling](https://mlsanity.com/paper/2207.06766/) (2022)
- [Towards Unbiased Label Distribution Learning for Facial Pose Estimation Using Anisotropic Spherical Gaussian](https://mlsanity.com/paper/2208.09122/) (2022)
- [HOTVCOM: Generating Buzzworthy Comments for Videos](https://mlsanity.com/paper/2409.15196/) (2024)
- [SEM-POS: Grammatically and Semantically Correct Video Captioning](https://mlsanity.com/paper/2303.14829/) (2023)
- [Learning Dynamic Query Combinations for Transformer-based Object Detection and Segmentation](https://mlsanity.com/paper/2307.12239/) (2023)
- [DFA: Dynamic Feature Aggregation for Efficient Video Object Detection](https://mlsanity.com/paper/2210.00588/) (2022)
- [Deep video representation learning: a survey](https://mlsanity.com/paper/2405.06574/) (2024)
- [Basketball-SORT: An Association Method for Complex Multi-object Occlusion Problems in Basketball Multi-object Tracking](https://mlsanity.com/paper/2406.19655/) (2024)
- [CodingHomo: Bootstrapping Deep Homography With Video Coding](https://mlsanity.com/paper/2504.12165/) (2025)
- [Towards Subject-Oriented Video Captioning via User-Specified Targets](https://mlsanity.com/paper/2312.13330/) (2023)
- [SSGA-Net: Stepwise Spatial Global-local Aggregation Networks for for Autonomous Driving](https://mlsanity.com/paper/2405.18857/) (2024)
- [The Staged Knowledge Distillation in Video Classification: Harmonizing Student Progress by a Complementary Weakly Supervised Framework](https://mlsanity.com/paper/2307.05201/) (2023)

## Abstract (arXiv)

Video captioning is a challenging task as it needs to accurately transform visual understanding into natural language description. To date, state-of-the-art methods inadequately model global-local representation across video frames for caption generation, leaving plenty of room for improvement. In this work, we approach the video captioning task from a new perspective and propose a GL-RG framework for video captioning, namely a Global-Local Representation Granularity. Our GL-RG demonstrates three advantages over the prior efforts: 1) we explicitly exploit extensive visual representations from different video ranges to improve linguistic expression; 2) we devise a novel global-local encoder to produce rich semantic vocabulary to obtain a descriptive granularity of video contents across frames; 3) we develop an incremental training strategy which organizes model learning in an incremental fashion to incur an optimal captioning behavior. Experimental results on the challenging MSR-VTT and MSVD datasets show that our DL-RG outperforms recent state-of-the-art methods by a significant margin. Code is available at urlhttps://github.com/ylqi/GL-RG.