mlsanityData to 14 Oct 2026

FTII-Bench: A Comprehensive Multimodal Benchmark for Flow Text with Image Insertion

arXiv 2410.12564 ↗First posted 16 Oct 2024cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Benefiting from the revolutionary advances in large language models (LLMs) and foundational vision models, large vision-language models (LVLMs) have also made significant progress. However, current benchmarks focus on tasks that evaluating only a single aspect of LVLM capabilities (e.g., recognition, detection, understanding). These tasks fail to fully demonstrate LVLMs' potential in complex application scenarios. To comprehensively assess the performance of existing LVLMs, we propose a more challenging task called the Flow Text with Image Insertion task (FTII). This task requires LVLMs to simultaneously possess outstanding abilities in image comprehension, instruction understanding, and long-text interpretation. Specifically, given several text paragraphs and a set of candidate images, as the text paragraphs accumulate, the LVLMs are required to select the most suitable image from the candidates to insert after the corresponding paragraph. Constructing a benchmark for such a task is highly challenging, particularly in determining the sequence of flowing text and images. To address this challenge, we turn to professional news reports, which naturally contain a gold standard for image-text sequences. Based on this, we introduce the Flow Text with Image Insertion Benchmark (FTII-Bench), which includes 318 high-quality Chinese image-text news articles and 307 high-quality English image-text news articles, covering 10 different news domains. Using these 625 high-quality articles, we construct problems of two different types with multiple levels of difficulty. Furthermore, we establish two different evaluation pipelines based on the CLIP model and existing LVLMs. We evaluate 9 open-source and 2 closed-source LVLMs as well as 2 CLIP-based models. Results indicate that even the most advanced models (e.g., GPT-4o) face significant challenges when tackling the FTII task.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 37 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

23 of 37 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024MM-CamObj: A Comprehensive Multimodal Dataset for Camouflaged Object Scenarios24 Sep
  2. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phone3 Aug
  3. 2024MANTIS: Interleaved Multi-Image Instruction Tuning2 May
  4. 2024BLINK: Multimodal Large Language Models Can See but Not Perceive18 Apr
  5. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 17

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  7. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  8. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  9. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  10. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  11. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  12. 2021Florence: A New Foundation Model for Computer Visioncited by 804
5 more
  1. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  2. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 123
  3. 2020TAP: Text-Aware Pre-training for Text-VQA and Text-Captioncited by 95
  4. 2023LAMM: Label Alignment for Multi-Modal Prompt Learningcited by 15
  5. 2023GIST: Improving Parameter Efficient Fine Tuning via Knowledge Interactioncited by 8

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025ReviewAgents: Bridging the Gap Between Human and AI-Generated Paper Reviewscited by 32
  2. 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Modelscited by 19
  3. 2025M2IO-R1: An Efficient RL-Enhanced Reasoning Framework for Multimodal Retrieval Augmented Multimodal Generationcited by 7

Similar, not cited

Close in wording, no citation either way

  1. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review
  2. 2026Deeply Interleaved Text-Image Contexts for Multimodal LLMs Assessment
  3. 2024II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Models
  4. 2025ReadBench: Measuring the Dense Text Visual Reading Ability of Vision-Language Models
  5. 2024SEED-Bench-2-Plus: Benchmarking Multimodal Large Language Models with Text-Rich Visual Comprehension
  6. 2025Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
  7. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  8. 2025IV-Bench: A Benchmark for Image-Grounded Video Perception and Reasoning in Multimodal LLMs