mlsanityData to 13 Oct 2026

Comparison Visual Instruction Tuning

arXiv 2406.09240 ↗First posted 13 Jun 2024cs.CVcited by 8

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Comparing two images in terms of Commonalities and Differences (CaD) is a fundamental human capability that forms the basis of advanced visual reasoning and interpretation. It is essential for the generation of detailed and contextually relevant descriptions, performing comparative analysis, novelty detection, and making informed decisions based on visual data. However, surprisingly, little attention has been given to these fundamental concepts in the best current mimic of human visual intelligence - Large Multimodal Models (LMMs). We develop and contribute a new two-phase approach CaD-VI for collecting synthetic visual instructions, together with an instruction-following dataset CaD-Inst containing 349K image pairs with CaD instructions collected using CaD-VI. Our approach significantly improves the CaD spotting capabilities in LMMs, advancing the SOTA on a diverse set of related tasks by up to 17.5%. It is also complementary to existing difference-only instruction datasets, allowing automatic targeted refinement of those resources increasing their effectiveness for CaD tuning by up to 10%. Additionally, we propose an evaluation benchmark with 7.5K open-ended QAs to assess the CaD understanding abilities of LMMs.

Key facts. Cited by 8 papers in the map (data through 13 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 51 of its 69 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

51 of 69 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 50

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,777
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,550
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  7. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  8. 2023Visual Instruction Tuningcited by 7,827
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,103
  10. 2023Improved Baselines with Visual Instruction Tuningcited by 4,389
  11. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,864
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,703
38 more
  1. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,649
  2. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,506
  3. 2024Mixtral of Expertscited by 1,811
  4. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,637
  5. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,452
  6. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,093
  7. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,068
  8. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 933
  9. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 792
  10. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 778
  11. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 689
  12. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 680
  13. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  14. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 563
  15. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  16. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 544
  17. 2020Rescaling Egocentric Visioncited by 532
  18. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  19. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  20. 2022Kubric: A scalable dataset generatorcited by 375
  21. 2024InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Modelcited by 326
  22. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  23. 2023LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understandingcited by 273
  24. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 265
  25. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  26. 2019Action Genome: Actions as Composition of Spatio-temporal Scene Graphscited by 249
  27. 2019Connecting Vision and Language with Localized Narrativescited by 225
  28. 2023MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learningcited by 171
  29. 2023SVIT: Scaling up Visual Instruction Tuningcited by 162
  30. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  31. 2018Learning to Describe Differences Between Pairs of Similar Imagescited by 114
  32. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  33. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 96
  34. 2023COLA: A Benchmark for Compositional Text-to-image Retrievalcited by 53
  35. 2023Equivariant Similarity for Vision-Language Foundation Modelscited by 51
  36. 2023Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Modelscited by 28
  37. 2022GEB+: A Benchmark for Generic Event Boundary Captioning, Grounding and Retrievalcited by 19
  38. 2019Evaluating Text-to-Image Matching using Binary Image Selection (BISON)cited by 12

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 1.

8 papers.

  1. 2024LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Contentcited by 17
  2. 2025MLLM-Selector: Necessity and Diversity-driven High-Value Data Selection for Enhanced Visual Instruction Tuningcited by 9
  3. 2025TTRV: Test-Time Reinforcement Learning for Vision Language Modelscited by 9
  4. 2024GLOV: Guided Large Language Models as Implicit Optimizers for Vision Language Modelscited by 6
  5. 2025PRISMM-Bench: A Benchmark of Peer-Review Grounded Multimodal Inconsistenciescited by 4
  6. 2024It's Just Another Day: Unique Video Captioning by Discriminative Promptingcited by 3
  7. 2025What's in Common? Multimodal Models Hallucinate When Reasoning Across Scenescited by 0
  8. 2026StochasT: Learning with Stochastic Turn Depth for Visual Instruction Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
  2. 2025RCI: A Score for Evaluating Global and Local Reasoning in Multimodal Benchmarks
  3. 2023An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
  4. 2024Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Model
  5. 2024How to Determine the Preferred Image Distribution of a Black-Box Vision-Language Model?
  6. 2025Picking the Cream of the Crop: Visual-Centric Data Selection with Collaborative Agents
  7. 2024HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Tasks
  8. 2023MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning