mlsanityData to 14 Oct 2026

ContextQFormer: A New Context Modeling Method for Multi-Turn Multi-Modal Conversations

arXiv 2505.23121 ↗First posted 29 May 2025cs.CL cs.AIcited by 3

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multi-modal large language models have demonstrated remarkable zero-shot abilities and powerful image-understanding capabilities. However, the existing open-source multi-modal models suffer from the weak capability of multi-turn interaction, especially for long contexts. To address the issue, we first introduce a context modeling module, termed ContextQFormer, which utilizes a memory block to enhance the presentation of contextual information. Furthermore, to facilitate further research, we carefully build a new multi-turn multi-modal dialogue dataset (TMDialog) for pre-training, instruction-tuning, and evaluation, which will be open-sourced lately. Compared with other multi-modal dialogue datasets, TMDialog contains longer conversations, which supports the research of multi-turn multi-modal dialogue. In addition, ContextQFormer is compared with three baselines on TMDialog and experimental results illustrate that ContextQFormer achieves an improvement of 2%-4% in available rate over baselines.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 37 references that are in the map. First posted on arXiv 29 May 2025.

Builds on

22 of 37 references.

References · 22

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,850
  7. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  8. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  10. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  11. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  12. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
10 more
  1. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  2. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  3. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  4. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  5. 2016Visual Dialogcited by 377
  6. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 356
  7. 2019Connecting Vision and Language with Localized Narrativescited by 226
  8. 2020Towards Conversational Recommendation over Multi-Type Dialogscited by 107
  9. 2022MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversationcited by 56
  10. 2021PhotoChat: A Human-Human Dialogue Dataset with Photo Sharing Behavior for Joint Image-Text Modelingcited by 43

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 126
  2. 2026Multi-turn Conversational AI from Text to Multimodal Interaction: Data, Models, Evaluation, and Open Challengescited by 1
  3. 2026StochasT: Learning with Stochastic Turn Depth for Visual Instruction Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MMCR: Advancing Visual Language Model in Multimodal Multi-Turn Contextual Reasoning
  2. 2025ContextualLVLM-Agent: A Holistic Framework for Multi-Turn Visually-Grounded Dialogue and Complex Instruction Following
  3. 2026Context-Agent: Dynamic Discourse Trees for Non-Linear Dialogue
  4. 2026Beyond Continuity: Challenges of Context Switching in Multi-Turn Dialogue with LLMs
  5. 2026ChatUMM: Robust Context Tracking for Conversational Interleaved Generation
  6. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integration
  7. 2025MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversation
  8. 2025Taking Notes Brings Focus? Towards Multi-Turn Multimodal Dialogue Learning