mlsanityData to 14 Oct 2026

MANTIS: Interleaved Multi-Image Instruction Tuning

arXiv 2405.01483 ↗First posted 2 May 2024cs.CV cs.AI cs.CLcited by 216

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Large multimodal models (LMMs) have shown great results in single-image vision language tasks. However, their abilities to solve multi-image visual language tasks is yet to be improved. The existing LMMs like OpenFlamingo, Emu2, and Idefics gain their multi-image ability through pre-training on hundreds of millions of noisy interleaved image-text data from the web, which is neither efficient nor effective. In this paper, we aim to build strong multi-image LMMs via instruction tuning with academic-level resources. Therefore, we meticulously construct Mantis-Instruct containing 721K multi-image instruction data to train a family of Mantis models. The instruction tuning empowers Mantis with different multi-image skills like co-reference, comparison, reasoning, and temporal understanding. We evaluate Mantis on 8 multi-image benchmarks and 6 single-image benchmarks. Mantis-Idefics2 can achieve SoTA results on all the multi-image benchmarks and beat the strongest multi-image baseline, Idefics2-8B by an average of 13 absolute points. Notably, Idefics2-8B was pre-trained on 140M interleaved multi-image data, which is 200x larger than Mantis-Instruct. We observe that Mantis performs equivalently well on the held-in and held-out benchmarks, which shows its generalization ability. We further evaluate Mantis on single-image benchmarks and demonstrate that Mantis also maintains a strong single-image performance on par with CogVLM and Emu2. Our results show that multi-image abilities are not necessarily gained through massive pre-training, instead, they can be gained by low-cost instruction tuning. The training and evaluation of Mantis has paved the road for future work to improve LMMs' multi-image abilities.

Key facts. Cited by 216 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 72 references that are in the map. First posted on arXiv 2 May 2024.

Builds on

53 of 72 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024MileBench: Benchmarking MLLMs in Long Context29 Apr
  2. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  3. 2024Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models18 Apr
  4. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training14 Mar
  5. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling19 Feb
  6. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences19 Jan
  7. 2023Generative Multimodal Models are In-Context Learners20 Dec
  8. 2023VILA: On Pre-training for Visual Language Models12 Dec
  9. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captions21 Nov
  10. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  11. 2023NExT-GPT: Any-to-Any Multimodal LLM11 Sep
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
2 more
  1. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  2. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  6. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  8. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  9. 2015VQA: Visual Question Answeringcited by 2,506
  10. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  11. 2019Towards VQA Models That Can Readcited by 1,718
  12. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
27 more
  1. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  2. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  3. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  4. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  5. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,109
  6. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  7. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 758
  8. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
  9. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  10. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  11. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 544
  12. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 459
  13. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  14. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  15. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 340
  16. 2023DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Datacited by 337
  17. 2023MIMIC-IT: Multi-Modal In-Context Instruction Tuningcited by 267
  18. 2023Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Visioncited by 249
  19. 2023Multimodal C4: An Open, Billion-scale Corpus of Images Interleaved with Textcited by 208
  20. 2023VIEScore: Towards Explainable Metrics for Conditional Image Synthesis Evaluationcited by 201
  21. 2016Visual Storytellingcited by 176
  22. 2023GPT-4V(ision) as a Generalist Evaluator for Vision-Language Taskscited by 119
  23. 2018Learning to Describe Differences Between Pairs of Similar Imagescited by 114
  24. 2024Towards Open-ended Visual Quality Comparisoncited by 85
  25. 2023Large Language Models are Temporal and Causal Reasoners for Video Question Answeringcited by 51
  26. 2019Neural Naturalist: Generating Fine-Grained Image Comparisonscited by 36
  27. 2022Image Retrieval from Contextual Descriptionscited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 71; 2025: 90; 2026: 55.

216 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  2. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  3. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  4. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  5. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  6. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  7. 2024Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Modelscited by 329
  8. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  9. 2025Improving Video Generation with Human Feedbackcited by 284
  10. 2024VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generationcited by 197
  11. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 188
  12. 2024Building and better understanding vision-language models: insights and future directionscited by 173
38 more
  1. 2024Oryx MLLM: On-Demand Spatial-Temporal Understanding at Arbitrary Resolutioncited by 159
  2. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 134
  3. 2024LLaVA-Critic: Learning to Evaluate Multimodal Modelscited by 134
  4. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scalecited by 127
  5. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  6. 2025MiMo-VL Technical Reportcited by 115
  7. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 102
  8. 2024OmniBench: Towards The Future of Universal Omni-Language Modelscited by 102
  9. 2024MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Enginescited by 101
  10. 2024Unveiling Encoder-Free Vision-Language Modelscited by 94
  11. 2024MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMscited by 90
  12. 2025UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learningcited by 84
  13. 2024GenAI Arena: An Open Evaluation Platform for Generative Modelscited by 82
  14. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 76
  15. 2025MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMscited by 73
  16. 2024LongDocURL: a Comprehensive Multimodal Long Document Benchmark Integrating Understanding, Reasoning, and Locatingcited by 72
  17. 2024MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Modelscited by 67
  18. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Taskscited by 57
  19. 2024Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmarkcited by 55
  20. 2025OmniMMI: A Comprehensive Multi-modal Interaction Benchmark in Streaming Video Contextscited by 51
  21. 2025Interleave-VLA: Enhancing Robot Manipulation with Interleaved Image-Text Instructionscited by 51
  22. 2024MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokenscited by 50
  23. 2024Mitigating Object Hallucination via Concentric Causal Attentioncited by 49
  24. 2024SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretationcited by 48
  25. 2025VideoMind: A Chain-of-LoRA Agent for Temporal-Grounded Video Reasoningcited by 43
  26. 2024MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Modelscited by 42
  27. 2025VScan: Rethinking Visual Token Reduction for Efficient Large Vision-Language Modelscited by 41
  28. 2024MIBench: Evaluating Multimodal Large Language Models over Multiple Imagescited by 39
  29. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMscited by 39
  30. 2025Reinforced Visual Perception with Toolscited by 38
  31. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  32. 2025Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformerscited by 36
  33. 2025TUNA: Taming Unified Visual Representations for Native Unified Multimodal Modelscited by 34
  34. 2024MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Modelscited by 33
  35. 2024MLLM-CompBench: A Comparative Reasoning Benchmark for Multimodal LLMscited by 32
  36. 2025MV-MATH: Evaluating Multimodal Math Reasoning in Multi-Visual Contextscited by 32
  37. 2025Mavors: Multi-granularity Video Representation for Multimodal Large Language Modelcited by 32
  38. 2025CAFe: Unifying Representation and Generation with Contrastive-Autoregressive Finetuningcited by 31

50 most cited of 216 All 216 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023On the Performance of Multimodal Language Models
  2. 2023Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs
  3. 2023An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models
  4. 2023Reformulating Vision-Language Foundation Models and Datasets Towards Universal Multimodal Assistants
  5. 2025MANZANO: A Simple and Scalable Unified Multimodal Model with a Hybrid Vision Tokenizer
  6. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models
  7. 2024MLAN: Language-Based Instruction Tuning Preserves and Transfers Knowledge in Multimodal Language Models
  8. 2026Visual Instruction Tuning Aligns Modalities through Abstraction