# BI-MDRG: Bridging Image History in Multimodal Dialogue Response Generation

arXiv: https://arxiv.org/abs/2408.05926 · HTML: https://mlsanity.com/paper/2408.05926/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 50 references that are in the map.
- First posted on arXiv 12 Aug 2024.

## Builds on (most cited first, 30 of 42)

- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation](https://mlsanity.com/paper/2208.12242/) (2022)
- [Improved Techniques for Training GANs](https://mlsanity.com/paper/1606.03498/) (2016)
- [An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion](https://mlsanity.com/paper/2208.01618/) (2022)
- [Multi-Concept Customization of Text-to-Image Diffusion](https://mlsanity.com/paper/2212.04488/) (2022)
- [eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers](https://mlsanity.com/paper/2211.01324/) (2022)
- [DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation](https://mlsanity.com/paper/1911.00536/) (2019)
- [MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversations](https://mlsanity.com/paper/1810.02508/) (2018)
- [OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Models](https://mlsanity.com/paper/2308.01390/) (2023)
- [BLIP-Diffusion: Pre-trained Subject Representation for Controllable Text-to-Image Generation and Editing](https://mlsanity.com/paper/2305.14720/) (2023)
- [AnyDoor: Zero-shot Object-level Image Customization](https://mlsanity.com/paper/2307.09481/) (2023)
- [Visual Dialog](https://mlsanity.com/paper/1611.08669/) (2016)
- [Subject-Diffusion:Open Domain Personalized Text-to-Image Generation without Test-time Fine-tuning](https://mlsanity.com/paper/2307.11410/) (2023)
- [Grounding Language Models to Images for Multimodal Inputs and Outputs](https://mlsanity.com/paper/2301.13823/) (2023)
- [Audio-Visual Scene-Aware Dialog](https://mlsanity.com/paper/1901.09107/) (2019)
- ["This is my unicorn, Fluffy": Personalizing frozen vision-language representations](https://mlsanity.com/paper/2204.01694/) (2022)
- [Image Chat: Engaging Grounded Conversations](https://mlsanity.com/paper/1811.00945/) (2018)
- [Two Causal Principles for Improving Visual Dialog](https://mlsanity.com/paper/1911.10496/) (2019)
- [MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversation](https://mlsanity.com/paper/2211.05719/) (2022)
- [M3ED: Multi-modal Multi-scene Multi-label Emotional Dialogue Database](https://mlsanity.com/paper/2205.10237/) (2022)
- [PhotoChat: A Human-Human Dialogue Dataset with Photo Sharing Behavior for Joint Image-Text Modeling](https://mlsanity.com/paper/2108.01453/) (2021)
- [Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog](https://mlsanity.com/paper/1902.00579/) (2019)
- [Recursive Visual Attention in Visual Dialog](https://mlsanity.com/paper/1812.02664/) (2018)
- [Multimodal Dialogue Response Generation](https://mlsanity.com/paper/2110.08515/) (2021)

## Built on by (most cited first, 8 of 8)

- [TPC: Test-time Procrustes Calibration for Diffusion-based Human Image Animation](https://mlsanity.com/paper/2410.24037/) (2024)
- [MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generation](https://mlsanity.com/paper/2410.02130/) (2024)
- [Multi-Party Conversational Agents: A Survey](https://mlsanity.com/paper/2505.18845/) (2025)
- [Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Models](https://mlsanity.com/paper/2507.04976/) (2025)
- [Diffusion Augmented Retrieval: A Training-Free Approach to Interactive Text-to-Image Retrieval](https://mlsanity.com/paper/2501.15379/) (2025)
- [PDCR: Perception-Decomposed Confidence Reward for Vision-Language Reasoning](https://mlsanity.com/paper/2605.13467/) (2026)
- [GenIR: Generative Visual Feedback for Mental Image Retrieval](https://mlsanity.com/paper/2506.06220/) (2025)
- [Router-Suggest: Dynamic Routing for Multimodal Auto-Completion in Visually-Grounded Dialogs](https://mlsanity.com/paper/2601.05851/) (2026)

## Abstract (arXiv)

Multimodal Dialogue Response Generation (MDRG) is a recently proposed task where the model needs to generate responses in texts, images, or a blend of both based on the dialogue context. Due to the lack of a large-scale dataset specifically for this task and the benefits of leveraging powerful pre-trained models, previous work relies on the text modality as an intermediary step for both the image input and output of the model rather than adopting an end-to-end approach. However, this approach can overlook crucial information about the image, hindering 1) image-grounded text response and 2) consistency of objects in the image response. In this paper, we propose BI-MDRG that bridges the response generation path such that the image history information is utilized for enhanced relevance of text responses to the image content and the consistency of objects in sequential image responses. Through extensive experiments on the multimodal dialogue benchmark dataset, we show that BI-MDRG can effectively increase the quality of multimodal dialogue. Additionally, recognizing the gap in benchmark datasets for evaluating the image consistency in multimodal dialogue, we have created a curated set of 300 dialogues annotated to track object consistency across conversations.