# Factor Graph Attention

arXiv: https://arxiv.org/abs/1904.05880 · HTML: https://mlsanity.com/paper/1904.05880/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Apr 2019.

## Built on by (most cited first, 30 of 41)

- [VideoLLM-online: Online Video Large Language Model for Streaming Video](https://mlsanity.com/paper/2406.11816/) (2024)
- [LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models](https://mlsanity.com/paper/2306.09265/) (2023)
- [ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic](https://mlsanity.com/paper/2111.14447/) (2021)
- [Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation](https://mlsanity.com/paper/2309.16429/) (2023)
- [Visual Navigation with Spatial Attention](https://mlsanity.com/paper/2104.09807/) (2021)
- [Perceptual Score: What Data Modalities Does Your Model Perceive?](https://mlsanity.com/paper/2110.14375/) (2021)
- [Graph Transformers: A Survey](https://mlsanity.com/paper/2407.09777/) (2024)
- [AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generation](https://mlsanity.com/paper/2305.13050/) (2023)
- [UTC: A Unified Transformer with Inter-Task Contrastive Learning for Visual Dialog](https://mlsanity.com/paper/2205.00423/) (2022)
- [Multimodal Conversational AI: A Survey of Datasets and Approaches](https://mlsanity.com/paper/2205.06907/) (2022)
- [Are VQA Systems RAD? Measuring Robustness to Augmented Data with Focused Interventions](https://mlsanity.com/paper/2106.04484/) (2021)
- [DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue](https://mlsanity.com/paper/2101.00151/) (2021)
- [GoG: Relation-aware Graph-over-Graph Network for Visual Dialog](https://mlsanity.com/paper/2109.08475/) (2021)
- [Latent Space Explanation by Intervention](https://mlsanity.com/paper/2112.04895/) (2021)
- [Learning Distinct and Representative Styles for Image Captioning](https://mlsanity.com/paper/2209.08231/) (2022)
- [Learning Reasoning Paths over Semantic Graphs for Video-grounded Dialogues](https://mlsanity.com/paper/2103.00820/) (2021)
- [VGNMN: Video-grounded Neural Module Network to Video-Grounded Language Tasks](https://mlsanity.com/paper/2104.07921/) (2021)
- [Classification-Regression for Chart Comprehension](https://mlsanity.com/paper/2111.14792/) (2021)
- [Learning to Ground Visual Objects for Visual Dialog](https://mlsanity.com/paper/2109.06013/) (2021)
- [Video and Text Matching with Conditioned Embeddings](https://mlsanity.com/paper/2110.11298/) (2021)
- [Modeling Text-visual Mutual Dependency for Multi-modal Dialog Generation](https://mlsanity.com/paper/2105.14445/) (2021)
- [Unified Questioner Transformer for Descriptive Question Generation in Goal-Oriented Visual Dialogue](https://mlsanity.com/paper/2106.15550/) (2021)
- [The Dialog Must Go On: Improving Visual Dialog via Generative Self-Training](https://mlsanity.com/paper/2205.12502/) (2022)
- [ReALM: Reference Resolution As Language Modeling](https://mlsanity.com/paper/2403.20329/) (2024)
- [Ordered Attention for Coherent Visual Storytelling](https://mlsanity.com/paper/2108.02180/) (2021)
- [Multimodal Dialogue State Tracking](https://mlsanity.com/paper/2206.07898/) (2022)
- [Ensemble of MRR and NDCG models for Visual Dialog](https://mlsanity.com/paper/2104.07511/) (2021)
- [Saying the Unseen: Video Descriptions via Dialog Agents](https://mlsanity.com/paper/2106.14069/) (2021)
- [Improving Cross-Modal Understanding in Visual Dialog via Contrastive Learning](https://mlsanity.com/paper/2204.07302/) (2022)
- [Unified Multimodal Model with Unlikelihood Training for Visual Dialog](https://mlsanity.com/paper/2211.13235/) (2022)

## Abstract (arXiv)

Dialog is an effective way to exchange information, but subtle details and nuances are extremely important. While significant progress has paved a path to address visual dialog with algorithms, details and nuances remain a challenge. Attention mechanisms have demonstrated compelling results to extract details in visual question answering and also provide a convincing framework for visual dialog due to their interpretability and effectiveness. However, the many data utilities that accompany visual dialog challenge existing attention techniques. We address this issue and develop a general attention mechanism for visual dialog which operates on any number of data utilities. To this end, we design a factor graph based attention mechanism which combines any number of utility representations. We illustrate the applicability of the proposed approach on the challenging and recently introduced VisDial datasets, outperforming recent state-of-the-art methods by 1.1% for VisDial0.9 and by 2% for VisDial1.0 on MRR. Our ensemble model improved the MRR score on VisDial1.0 by more than 6%.