# Dual Attention Networks for Multimodal Reasoning and Matching

arXiv: https://arxiv.org/abs/1611.00471 · HTML: https://mlsanity.com/paper/1611.00471/

## Key facts

- Cited by 114 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Nov 2016.

## Built on by (most cited first, 30 of 114)

- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Focal Self-attention for Local-Global Interactions in Vision Transformers](https://mlsanity.com/paper/2107.00641/) (2021)
- [ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias](https://mlsanity.com/paper/2106.03348/) (2021)
- [Combined Scaling for Zero-shot Transfer Learning](https://mlsanity.com/paper/2111.10050/) (2021)
- [Probabilistic Embeddings for Cross-Modal Retrieval](https://mlsanity.com/paper/2101.05068/) (2021)
- [Similarity Reasoning and Filtration for Image-Text Matching](https://mlsanity.com/paper/2101.01368/) (2021)
- [Vision-Language Pre-training: Basics, Recent Advances, and Future Trends](https://mlsanity.com/paper/2210.09263/) (2022)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [DSSL: Deep Surroundings-person Separation Learning for Text-based Person Retrieval](https://mlsanity.com/paper/2109.05534/) (2021)
- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [Adaptive Multi-Teacher Multi-level Knowledge Distillation](https://mlsanity.com/paper/2103.04062/) (2021)
- [Bridge to Answer: Structure-aware Graph Interaction Network for Video Question Answering](https://mlsanity.com/paper/2104.14085/) (2021)
- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Bi-Temporal Semantic Reasoning for the Semantic Change Detection in HR Remote Sensing Images](https://mlsanity.com/paper/2108.06103/) (2021)
- [BenchLMM: Benchmarking Cross-style Visual Capability of Large Multimodal Models](https://mlsanity.com/paper/2312.02896/) (2023)
- [CAMS: An Annotated Corpus for Causal Analysis of Mental Health Issues in Social Media Posts](https://mlsanity.com/paper/2207.04674/) (2022)
- [PROSE-FD: A Multimodal PDE Foundation Model for Learning Multiple Operators for Forecasting Fluid Dynamics](https://mlsanity.com/paper/2409.09811/) (2024)
- [Improving Cross-Modal Retrieval with Set of Diverse Embeddings](https://mlsanity.com/paper/2211.16761/) (2022)
- [Spatio-Temporal Representation Factorization for Video-based Person Re-Identification](https://mlsanity.com/paper/2107.11878/) (2021)
- [Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval](https://mlsanity.com/paper/2103.11920/) (2021)
- [Vision-Language Intelligence: Tasks, Representation Learning, and Large Models](https://mlsanity.com/paper/2203.01922/) (2022)
- [Beyond Fixation: Dynamic Window Visual Transformer](https://mlsanity.com/paper/2203.12856/) (2022)
- [Looking Outside the Window: Wide-Context Transformer for the Semantic Segmentation of High-Resolution Remote Sensing Images](https://mlsanity.com/paper/2106.15754/) (2021)
- [Multi-modal Machine Learning in Engineering Design: A Review and Future Directions](https://mlsanity.com/paper/2302.10909/) (2023)
- [Step-Wise Hierarchical Alignment Network for Image-Text Matching](https://mlsanity.com/paper/2106.06509/) (2021)
- [Multimodal Explainable Artificial Intelligence: A Comprehensive Review of Methodological Advances and Future Research Directions](https://mlsanity.com/paper/2306.05731/) (2023)
- [From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities](https://mlsanity.com/paper/2311.00308/) (2023)
- [Multimodal Pre-training Based on Graph Attention Network for Document Understanding](https://mlsanity.com/paper/2203.13530/) (2022)
- [Structured Multi-modal Feature Embedding and Alignment for Image-Sentence Retrieval](https://mlsanity.com/paper/2108.02417/) (2021)

## Abstract (arXiv)

We propose Dual Attention Networks (DANs) which jointly leverage visual and textual attention mechanisms to capture fine-grained interplay between vision and language. DANs attend to specific regions in images and words in text through multiple steps and gather essential information from both modalities. Based on this framework, we introduce two types of DANs for multimodal reasoning and matching, respectively. The reasoning model allows visual and textual attentions to steer each other during collaborative inference, which is useful for tasks such as Visual Question Answering (VQA). In addition, the matching model exploits the two attention mechanisms to estimate the similarity between images and sentences by focusing on their shared semantics. Our extensive experiments validate the effectiveness of DANs in combining vision and language, achieving the state-of-the-art performance on public benchmarks for VQA and image-text matching.