# Object Relational Graph with Teacher-Recommended Learning for Video Captioning

arXiv: https://arxiv.org/abs/2002.11566 · HTML: https://mlsanity.com/paper/2002.11566/

## Key facts

- Cited by 107 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Feb 2020.

## Built on by (most cited first, 30 of 107)

- [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [TransVG: End-to-End Visual Grounding with Transformers](https://mlsanity.com/paper/2104.08541/) (2021)
- [Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning](https://mlsanity.com/paper/2302.14115/) (2023)
- [Any-to-Any Generation via Composable Diffusion](https://mlsanity.com/paper/2305.11846/) (2023)
- [SwinBERT: End-to-End Transformers with Sparse Attention for Video Captioning](https://mlsanity.com/paper/2111.13196/) (2021)
- [VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset](https://mlsanity.com/paper/2304.08345/) (2023)
- [Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks](https://mlsanity.com/paper/2112.01522/) (2021)
- [End-to-end Generative Pretraining for Multimodal Video Captioning](https://mlsanity.com/paper/2201.08264/) (2022)
- [Self-Supervised Learning of Object Parts for Semantic Segmentation](https://mlsanity.com/paper/2204.13101/) (2022)
- [CLIP4Caption: CLIP for Video Caption](https://mlsanity.com/paper/2110.06615/) (2021)
- [Learning Audio-Video Modalities from Image Captions](https://mlsanity.com/paper/2204.00679/) (2022)
- [ADAPT: Action-aware Driving Caption Transformer](https://mlsanity.com/paper/2302.00673/) (2023)
- [A Survey on Long-Tailed Visual Recognition](https://mlsanity.com/paper/2205.13775/) (2022)
- [VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captioners](https://mlsanity.com/paper/2212.04979/) (2022)
- [CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation](https://mlsanity.com/paper/2311.18775/) (2023)
- [Joint Inductive and Transductive Learning for Video Object Segmentation](https://mlsanity.com/paper/2108.03679/) (2021)
- [Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEs](https://mlsanity.com/paper/2206.04674/) (2022)
- [Expectation-Maximization Contrastive Learning for Compact Video-and-Language Representations](https://mlsanity.com/paper/2211.11427/) (2022)
- [A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective](https://mlsanity.com/paper/2209.13232/) (2022)
- [Exploring Sequence Feature Alignment for Domain Adaptive Detection Transformers](https://mlsanity.com/paper/2107.12636/) (2021)
- [Open-book Video Captioning with Retrieve-Copy-Generate Network](https://mlsanity.com/paper/2103.05284/) (2021)
- [Learning Conditional Knowledge Distillation for Degraded-Reference Image Quality Assessment](https://mlsanity.com/paper/2108.07948/) (2021)
- [VidChapters-7M: Video Chapters at Scale](https://mlsanity.com/paper/2309.13952/) (2023)
- [Disentangle Your Dense Object Detector](https://mlsanity.com/paper/2107.02963/) (2021)
- [Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectives](https://mlsanity.com/paper/2406.05615/) (2024)
- [EMScore: Evaluating Video Captioning via Coarse-Grained and Fine-Grained Embedding Matching](https://mlsanity.com/paper/2111.08919/) (2021)
- [GPT4Video: A Unified Multimodal Large Language Model for lnstruction-Followed Understanding and Safety-Aware Generation](https://mlsanity.com/paper/2311.16511/) (2023)
- [Semantic Grouping Network for Video Captioning](https://mlsanity.com/paper/2102.00831/) (2021)
- [Hierarchical Modular Network for Video Captioning](https://mlsanity.com/paper/2111.12476/) (2021)

## Abstract (arXiv)

Taking full advantage of the information from both vision and language is critical for the video captioning task. Existing models lack adequate visual representation due to the neglect of interaction between object, and sufficient training for content-related words due to long-tailed problems. In this paper, we propose a complete video captioning system including both a novel model and an effective training strategy. Specifically, we propose an object relational graph (ORG) based encoder, which captures more detailed interaction features to enrich visual representation. Meanwhile, we design a teacher-recommended learning (TRL) method to make full use of the successful external language model (ELM) to integrate the abundant linguistic knowledge into the caption model. The ELM generates more semantically similar word proposals which extend the ground-truth words used for training to deal with the long-tailed problem. Experimental evaluations on three benchmarks: MSVD, MSR-VTT and VATEX show the proposed ORG-TRL system achieves state-of-the-art performance. Extensive ablation studies and visualizations illustrate the effectiveness of our system.