# High-Order Attention Models for Visual Question Answering

arXiv: https://arxiv.org/abs/1711.04323 · HTML: https://mlsanity.com/paper/1711.04323/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Nov 2017.

## Built on by (most cited first, 22 of 22)

- [ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic](https://mlsanity.com/paper/2111.14447/) (2021)
- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [MeaCap: Memory-Augmented Zero-shot Image Captioning](https://mlsanity.com/paper/2403.03715/) (2024)
- [ConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishing](https://mlsanity.com/paper/2303.02437/) (2023)
- [Multi-modal Machine Learning in Engineering Design: A Review and Future Directions](https://mlsanity.com/paper/2302.10909/) (2023)
- [Zero-Shot Video Captioning with Evolving Pseudo-Tokens](https://mlsanity.com/paper/2207.11100/) (2022)
- [Dealing with Missing Modalities in the Visual Question Answer-Difference Prediction Task through Knowledge Distillation](https://mlsanity.com/paper/2104.05965/) (2021)
- [Latent Space Explanation by Intervention](https://mlsanity.com/paper/2112.04895/) (2021)
- [What is Where by Looking: Weakly-Supervised Open-World Phrase-Grounding without Text Inputs](https://mlsanity.com/paper/2206.09358/) (2022)
- [Video and Text Matching with Conditioned Embeddings](https://mlsanity.com/paper/2110.11298/) (2021)
- [A Comprehensive Survey of 3D Dense Captioning: Localizing and Describing Objects in 3D Scenes](https://mlsanity.com/paper/2403.07469/) (2024)
- [A Comprehensive Survey on Visual Question Answering Datasets and Algorithms](https://mlsanity.com/paper/2411.11150/) (2024)
- [Ordered Attention for Coherent Visual Storytelling](https://mlsanity.com/paper/2108.02180/) (2021)
- [QLEVR: A Diagnostic Dataset for Quantificational Language and Elementary Visual Reasoning](https://mlsanity.com/paper/2205.03075/) (2022)
- [AlignVE: Visual Entailment Recognition Based on Alignment Relations](https://mlsanity.com/paper/2211.08736/) (2022)
- [Tensor Sketch: Fast and Scalable Polynomial Kernel Approximation](https://mlsanity.com/paper/2505.08146/) (2025)
- [Dual Decomposition of Convex Optimization Layers for Consistent Attention in Medical Images](https://mlsanity.com/paper/2206.02761/) (2022)
- [Efficient Human Pose Estimation by Maximizing Fusion and High-Level Spatial Attention](https://mlsanity.com/paper/2107.13693/) (2021)
- [Semantic Segmentation Enhanced Transformer Model for Human Attention Prediction](https://mlsanity.com/paper/2301.11022/) (2023)
- [Monotone deep Boltzmann machines](https://mlsanity.com/paper/2307.04990/) (2023)
- [From text to multimodal: a survey of adversarial example generation in question answering systems](https://mlsanity.com/paper/2312.16156/) (2023)
- [Social Media Ready Caption Generation for Brands](https://mlsanity.com/paper/2401.01637/) (2024)

## Abstract (arXiv)

The quest for algorithms that enable cognitive abilities is an important part of machine learning. A common trait in many recently investigated cognitive-like tasks is that they take into account different data modalities, such as visual and textual input. In this paper we propose a novel and generally applicable form of attention mechanism that learns high-order correlations between various data modalities. We show that high-order correlations effectively direct the appropriate attention to the relevant elements in the different data modalities that are required to solve the joint task. We demonstrate the effectiveness of our high-order attention mechanism on the task of visual question answering (VQA), where we achieve state-of-the-art performance on the standard VQA dataset.