# Object-aware Aggregation with Bidirectional Temporal Graph for Video Captioning

arXiv: https://arxiv.org/abs/1906.04375 · HTML: https://mlsanity.com/paper/1906.04375/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Jun 2019.

## Built on by (most cited first, 30 of 42)

- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning](https://mlsanity.com/paper/2111.13196/) (2021)
- [End-to-end Generative Pretraining for Multimodal Video Captioning](https://mlsanity.com/paper/2201.08264/) (2022)
- [Open-book Video Captioning with Retrieve-Copy-Generate Network](https://mlsanity.com/paper/2103.05284/) (2021)
- [Semantic Grouping Network for Video Captioning](https://mlsanity.com/paper/2102.00831/) (2021)
- [Hierarchical Modular Network for Video Captioning](https://mlsanity.com/paper/2111.12476/) (2021)
- [Text with Knowledge Graph Augmented Transformer for Video Captioning](https://mlsanity.com/paper/2303.12423/) (2023)
- [A Review of Deep Learning for Video Captioning](https://mlsanity.com/paper/2304.11431/) (2023)
- [OmniVid: A Generative Framework for Universal Video Understanding](https://mlsanity.com/paper/2403.17935/) (2024)
- [O2NA: An Object-Oriented Non-Autoregressive Approach for Controllable Video Captioning](https://mlsanity.com/paper/2108.02359/) (2021)
- [Refined Semantic Enhancement towards Frequency Diffusion for Video Captioning](https://mlsanity.com/paper/2211.15076/) (2022)
- [Towards Efficient Partially Relevant Video Retrieval with Active Moment Discovering](https://mlsanity.com/paper/2504.10920/) (2025)
- [GL-RG: Global-Local Representation Granularity for Video Captioning](https://mlsanity.com/paper/2205.10706/) (2022)
- [MELTR: Meta Loss Transformer for Learning to Fine-tune Video Foundation Models](https://mlsanity.com/paper/2303.13009/) (2023)
- [Sports Video Analysis on Large-Scale Data](https://mlsanity.com/paper/2208.04897/) (2022)
- [Enrich and Detect: Video Temporal Grounding with Multimodal LLMs](https://mlsanity.com/paper/2510.17023/) (2025)
- [Discriminative Latent Semantic Graph for Video Captioning](https://mlsanity.com/paper/2108.03662/) (2021)
- [HOTVCOM: Generating Buzzworthy Comments for Videos](https://mlsanity.com/paper/2409.15196/) (2024)
- [IntentVCNet: Bridging Spatio-Temporal Gaps for Intention-Oriented Controllable Video Captioning](https://mlsanity.com/paper/2507.18531/) (2025)
- [SEM-POS: Grammatically and Semantically Correct Video Captioning](https://mlsanity.com/paper/2303.14829/) (2023)
- [Koopman Invertible Autoencoder: Leveraging Forward and Backward Dynamics for Temporal Modeling](https://mlsanity.com/paper/2309.10291/) (2023)
- [Boosting Video Representation Learning with Multi-Faceted Integration](https://mlsanity.com/paper/2201.04023/) (2022)
- [CREATE: A Benchmark for Chinese Short Video Retrieval and Title Generation](https://mlsanity.com/paper/2203.16763/) (2022)
- [Cross-Modal Graph with Meta Concepts for Video Captioning](https://mlsanity.com/paper/2108.06458/) (2021)
- [Visual Commonsense-aware Representation Network for Video Captioning](https://mlsanity.com/paper/2211.09469/) (2022)
- [UniBoost: Unsupervised Unimodal Pre-training for Boosting Zero-shot Vision-Language Tasks](https://mlsanity.com/paper/2306.04715/) (2023)
- [Video Captioning: a comparative review of where we are and which could be the route](https://mlsanity.com/paper/2204.05976/) (2022)
- [Boosting Video Captioning with Dynamic Loss Network](https://mlsanity.com/paper/2107.11707/) (2021)
- [Synchronized Audio-Visual Frames with Fractional Positional Encoding for Transformers in Video-to-Text Translation](https://mlsanity.com/paper/2112.14088/) (2021)
- [Revitalize Region Feature for Democratizing Video-Language Pre-training of Retrieval](https://mlsanity.com/paper/2203.07720/) (2022)

## Abstract (arXiv)

Video captioning aims to automatically generate natural language descriptions of video content, which has drawn a lot of attention recent years. Generating accurate and fine-grained captions needs to not only understand the global content of video, but also capture the detailed object information. Meanwhile, video representations have great impact on the quality of generated captions. Thus, it is important for video captioning to capture salient objects with their detailed temporal dynamics, and represent them using discriminative spatio-temporal representations. In this paper, we propose a new video captioning approach based on object-aware aggregation with bidirectional temporal graph (OA-BTG), which captures detailed temporal dynamics for salient objects in video, and learns discriminative spatio-temporal representations by performing object-aware local feature aggregation on detected object regions. The main novelties and advantages are: (1) Bidirectional temporal graph: A bidirectional temporal graph is constructed along and reversely along the temporal order, which provides complementary ways to capture the temporal trajectories for each salient object. (2) Object-aware aggregation: Learnable VLAD (Vector of Locally Aggregated Descriptors) models are constructed on object temporal trajectories and global frame sequence, which performs object-aware aggregation to learn discriminative representations. A hierarchical attention mechanism is also developed to distinguish different contributions of multiple objects. Experiments on two widely-used datasets demonstrate our OA-BTG achieves state-of-the-art performance in terms of BLEU@4, METEOR and CIDEr metrics.