# SMIR: Efficient Synthetic Data Pipeline To Improve Multi-Image Reasoning

arXiv: https://arxiv.org/abs/2501.03675 · HTML: https://mlsanity.com/paper/2501.03675/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 30 of its 46 references that are in the map.
- First posted on arXiv 7 Jan 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)
- Synthetic data and self-improvement for LLMs / [Self-improving MLLMs and synthetic data](https://mlsanity.com/synth/self-improving-mllms-and-synthetic-data/)

## Builds on (most cited first, 30 of 30)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [Instruction Tuning with GPT-4](https://mlsanity.com/paper/2304.03277/) (2023)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models](https://mlsanity.com/paper/2311.17043/) (2023)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [What matters when building vision-language models?](https://mlsanity.com/paper/2405.02246/) (2024)
- [MIMIC-IT: Multi-Modal In-Context Instruction Tuning](https://mlsanity.com/paper/2306.05425/) (2023)
- [WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild](https://mlsanity.com/paper/2406.04770/) (2024)
- [MANTIS: Interleaved Multi-Image Instruction Tuning](https://mlsanity.com/paper/2405.01483/) (2024)
- [To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning](https://mlsanity.com/paper/2311.07574/) (2023)
- [Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models](https://mlsanity.com/paper/2403.00231/) (2024)
- [MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning](https://mlsanity.com/paper/2212.10773/) (2022)
- [MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs](https://mlsanity.com/paper/2406.11833/) (2024)
- [MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models](https://mlsanity.com/paper/2408.02718/) (2024)
- [MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens](https://mlsanity.com/paper/2406.11271/) (2024)
- [MIBench: Evaluating Multimodal Large Language Models over Multiple Images](https://mlsanity.com/paper/2407.15272/) (2024)
- [MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity](https://mlsanity.com/paper/2407.15838/) (2024)
- [ConvBench: A Multi-Turn Conversation Evaluation Benchmark with Hierarchical Capability for Large Vision-Language Models](https://mlsanity.com/paper/2403.20194/) (2024)

## Built on by (most cited first, 3 of 3)

- [Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding](https://mlsanity.com/paper/2601.10611/) (2026)
- [Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)](https://mlsanity.com/paper/2504.03151/) (2025)
- [Visual-TableQA: Open-Domain Benchmark for Reasoning over Table Images](https://mlsanity.com/paper/2509.07966/) (2025)

## Abstract (arXiv)

Vision-Language Models (VLMs) excel at understanding single images, aided by high-quality instruction datasets. However, multi-image reasoning remains underexplored in the open-source community due to two key challenges: (1) scaling datasets with correlated images and complex reasoning instructions is resource-intensive, and (2) robust evaluation benchmarks for multi-image tasks are lacking. To address this, we introduce SMiR, a synthetic data-generation pipeline for multi-image reasoning, along with a high-quality dataset generated using this pipeline. SMiR efficiently extracts correlated images via multimodal embeddings, integrates visual and descriptive information, and leverages open-source LLMs to generate quality instructions. Using this approach, we produce 160K synthetic training samples, offering a cost-effective alternative to closed-source solutions. Additionally, we present SMiR-Bench, a multi-image reasoning benchmark comprising 200 diverse examples across seven complex reasoning tasks. SMiR-Bench is multi-turn and employs a VLM judge to evaluate free-form responses, providing a comprehensive assessment of model expressiveness and reasoning capability across modalities. We demonstrate the effectiveness of SMiR by fine-tuning open-source VLMs and evaluating them on SMiR-Bench.