# Describing Unseen Videos via Multi-Modal Cooperative Dialog Agents

arXiv: https://arxiv.org/abs/2008.07935 · HTML: https://mlsanity.com/paper/2008.07935/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 Aug 2020.

## Built on by (most cited first, 11 of 11)

- [Learning to Answer Questions in Dynamic Audio-Visual Scenarios](https://mlsanity.com/paper/2203.14072/) (2022)
- [Discrete Contrastive Diffusion for Cross-Modal Music and Image Generation](https://mlsanity.com/paper/2206.07771/) (2022)
- [Vision+X: A Survey on Multimodal Learning in the Light of Data](https://mlsanity.com/paper/2210.02884/) (2022)
- [Target-Aware Spatio-Temporal Reasoning via Answering Questions in Dynamics Audio-Visual Scenarios](https://mlsanity.com/paper/2305.12397/) (2023)
- [A Metamodel and Framework for Artificial General Intelligence From Theory to Practice](https://mlsanity.com/paper/2102.06112/) (2021)
- [Saying the Unseen: Video Descriptions via Dialog Agents](https://mlsanity.com/paper/2106.14069/) (2021)
- [Hear Me, See Me, Understand Me: Audio-Visual Autism Behavior Recognition](https://mlsanity.com/paper/2406.02554/) (2024)
- [CLIP-Powered TASS: Target-Aware Single-Stream Network for Audio-Visual Question Answering](https://mlsanity.com/paper/2405.07451/) (2024)
- [SaSR-Net: Source-Aware Semantic Representation Network for Enhancing Audio-Visual Question Answering](https://mlsanity.com/paper/2411.04933/) (2024)
- [Win the Lottery Ticket via Fourier Analysis: Frequencies Guided Network Pruning](https://mlsanity.com/paper/2201.12712/) (2022)
- [Supplementing Missing Visions via Dialog for Scene Graph Generations](https://mlsanity.com/paper/2204.11143/) (2022)

## Abstract (arXiv)

With the arising concerns for the AI systems provided with direct access to abundant sensitive information, researchers seek to develop more reliable AI with implicit information sources. To this end, in this paper, we introduce a new task called video description via two multi-modal cooperative dialog agents, whose ultimate goal is for one conversational agent to describe an unseen video based on the dialog and two static frames. Specifically, one of the intelligent agents - Q-BOT - is given two static frames from the beginning and the end of the video, as well as a finite number of opportunities to ask relevant natural language questions before describing the unseen video. A-BOT, the other agent who has already seen the entire video, assists Q-BOT to accomplish the goal by providing answers to those questions. We propose a QA-Cooperative Network with a dynamic dialog history update learning mechanism to transfer knowledge from A-BOT to Q-BOT, thus helping Q-BOT to better describe the video. Extensive experiments demonstrate that Q-BOT can effectively learn to describe an unseen video by the proposed model and the cooperative learning method, achieving the promising performance where Q-BOT is given the full ground truth history dialog.