# Discourse Analysis for Evaluating Coherence in Video Paragraph Captions

arXiv: https://arxiv.org/abs/2201.06207 · HTML: https://mlsanity.com/paper/2201.06207/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 36 references that are in the map.
- First posted on arXiv 17 Jan 2022.

## Builds on (most cited first, 9 of 9)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks](https://mlsanity.com/paper/1510.07712/) (2015)
- [Words aren't enough, their order matters: On the Robustness of Grounding Visual Referring Expressions](https://mlsanity.com/paper/2005.01655/) (2020)
- [Natural Language Interaction with Explainable AI Models](https://mlsanity.com/paper/1903.05720/) (2019)
- [X-ToM: Explaining with Theory-of-Mind for Gaining Justified Human Trust](https://mlsanity.com/paper/1909.06907/) (2019)
- [Effective Representation to Capture Collaboration Behaviors between Explainer and User](https://mlsanity.com/paper/2201.03147/) (2022)

## Built on by (most cited first, 2 of 2)

- [Attention cannot be an Explanation](https://mlsanity.com/paper/2201.11194/) (2022)
- [Question Generation for Evaluating Cross-Dataset Shifts in Multi-modal Grounding](https://mlsanity.com/paper/2201.09639/) (2022)

## Abstract (arXiv)

Video paragraph captioning is the task of automatically generating a coherent paragraph description of the actions in a video. Previous linguistic studies have demonstrated that coherence of a natural language text is reflected by its discourse structure and relations. However, existing video captioning methods evaluate the coherence of generated paragraphs by comparing them merely against human paragraph annotations and fail to reason about the underlying discourse structure. At UCLA, we are currently exploring a novel discourse based framework to evaluate the coherence of video paragraphs. Central to our approach is the discourse representation of videos, which helps in modeling coherence of paragraphs conditioned on coherence of videos. We also introduce DisNet, a novel dataset containing the proposed visual discourse annotations of 3000 videos and their paragraphs. Our experiment results have shown that the proposed framework evaluates coherence of video paragraphs significantly better than all the baseline methods. We believe that many other multi-discipline Artificial Intelligence problems such as Visual Dialog and Visual Storytelling would also greatly benefit from the proposed visual discourse framework and the DisNet dataset.