# Ensemble of MRR and NDCG models for Visual Dialog

arXiv: https://arxiv.org/abs/2104.07511 · HTML: https://mlsanity.com/paper/2104.07511/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 24 references that are in the map.
- First posted on arXiv 15 Apr 2021.

## Builds on (most cited first, 19 of 19)

- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Visual Dialog](https://mlsanity.com/paper/1611.08669/) (2016)
- [How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation](https://mlsanity.com/paper/1603.08023/) (2016)
- [A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues](https://mlsanity.com/paper/1605.06069/) (2016)
- [Learning to Reason: End-to-End Module Networks for Visual Question Answering](https://mlsanity.com/paper/1704.05526/) (2017)
- [GuessWhat?! Visual object discovery through multi-modal dialogue](https://mlsanity.com/paper/1611.08481/) (2016)
- [Two Causal Principles for Improving Visual Dialog](https://mlsanity.com/paper/1911.10496/) (2019)
- [Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline](https://mlsanity.com/paper/1912.02379/) (2019)
- [VD-BERT: A Unified Vision and Dialog Transformer with BERT](https://mlsanity.com/paper/2004.13278/) (2020)
- [Visual Coreference Resolution in Visual Dialog using Neural Module Networks](https://mlsanity.com/paper/1809.01816/) (2018)
- [A Simple Baseline for Audio-Visual Scene-Aware Dialog](https://mlsanity.com/paper/1904.05876/) (2019)
- [Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog](https://mlsanity.com/paper/1902.00579/) (2019)
- [Factor Graph Attention](https://mlsanity.com/paper/1904.05880/) (2019)
- [Recursive Visual Attention in Visual Dialog](https://mlsanity.com/paper/1812.02664/) (2018)
- [DualVD: An Adaptive Dual Encoding Model for Deep Visual Understanding in Visual Dialogue](https://mlsanity.com/paper/1911.07251/) (2019)
- [Dual Attention Networks for Visual Reference Resolution in Visual Dialog](https://mlsanity.com/paper/1902.09368/) (2019)
- [Reasoning Visual Dialogs with Structural and Partial Observations](https://mlsanity.com/paper/1904.05548/) (2019)
- [Two can play this Game: Visual Dialog with Discriminative Question Generation and Answering](https://mlsanity.com/paper/1803.11186/) (2018)
- [Efficient Attention Mechanism for Visual Dialog that can Handle All the Interactions between Multiple Inputs](https://mlsanity.com/paper/1911.11390/) (2019)

## Built on by (most cited first, 5 of 5)

- [Multimodal Knowledge Alignment with Reinforcement Learning](https://mlsanity.com/paper/2205.12630/) (2022)
- [FinSage: A Multi-aspect RAG System for Financial Filings Question Answering](https://mlsanity.com/paper/2504.14493/) (2025)
- [This Prompt is Measuring <MASK>: Evaluating Bias Evaluation in Language Models](https://mlsanity.com/paper/2305.12757/) (2023)
- [LawPal : A Retrieval Augmented Generation Based System for Enhanced Legal Accessibility in India](https://mlsanity.com/paper/2502.16573/) (2025)
- [Multi-Modal Agents for Power Distribution Defect Detection: An Evaluation of Foundation Models](https://mlsanity.com/paper/2606.12969/) (2026)

## Abstract (arXiv)

Assessing an AI agent that can converse in human language and understand visual content is challenging. Generation metrics, such as BLEU scores favor correct syntax over semantics. Hence a discriminative approach is often used, where an agent ranks a set of candidate options. The mean reciprocal rank (MRR) metric evaluates the model performance by taking into account the rank of a single human-derived answer. This approach, however, raises a new challenge: the ambiguity and synonymy of answers, for instance, semantic equivalence (e.g., `yeah' and `yes'). To address this, the normalized discounted cumulative gain (NDCG) metric has been used to capture the relevance of all the correct answers via dense annotations. However, the NDCG metric favors the usually applicable uncertain answers such as `I don't know. Crafting a model that excels on both MRR and NDCG metrics is challenging. Ideally, an AI agent should answer a human-like reply and validate the correctness of any answer. To address this issue, we describe a two-step non-parametric ranking approach that can merge strong MRR and NDCG models. Using our approach, we manage to keep most MRR state-of-the-art performance (70.41% vs. 71.24%) and the NDCG state-of-the-art performance (72.16% vs. 75.35%). Moreover, our approach won the recent Visual Dialog 2020 challenge. Source code is available at https://github.com/idansc/mrr-ndcg.