mlsanityData to 14 Oct 2026

What Breaks in Multi-Image Schema Application? Factorial Oracle Interventions Reveal an Instance-Information Bottleneck

arXiv 2610.19611 ↗First posted 14 Oct 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG also fits: VLM benchmarks and visual reasoning

Abstract (arXiv)

Modern vision-language models can accept multi-image contexts, but aggregate accuracy rarely reveals where a multi-image reasoning pipeline fails. We study a controlled schema-application task in which a model must locate two target characters in a 16-image sequence, identify their visual attributes, consult an item-specific visual codebook, and compose two ordered nonce code names. We introduce a within-item 2-by-2 factorial Oracle design: Oracle-A supplies a verified textual schema, Oracle-B supplies a verified ordered target-appearance representation, and both interventions retain the original images and final question. Across 72 new items and 1,188 analyzed model responses, Oracle-B increased strict exact accuracy by 54.6 percentage points (95% confidence interval: 46.3 to 63.0) and outperformed Oracle-A by 60.6 points (Holm-adjusted p-value no greater than .0002). An auxiliary retrieval diagnostic reached only 49.1% ordered-appearance accuracy even after removing code-name generation. Providing both Oracle sources yielded 100% accuracy across all models. These results localize the dominant recoverable deficit to upstream ordered instance recovery rather than primarily to downstream schema use and composition.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 44 references that are in the map. First posted on arXiv 14 Oct 2026.

Builds on

38 of 44 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Visual Haystacks: A Vision-Centric Needle-In-A-Haystack Benchmark18 Jul

Other references · 37

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  2. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  3. 2024MANTIS: Interleaved Multi-Image Instruction Tuningcited by 216
  4. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 188
  5. 2024Premise Order Matters in Reasoning with Large Language Modelscited by 72
  6. 2024Needle In A Multimodal Haystackcited by 58
  7. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Modelscited by 55
  8. 2024Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoningcited by 54
  9. 2026From Agent Traces to Trust: A Survey of Evidence Tracing and Execution Provenance in LLM Agentscited by 37
  10. 2026SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Modelscited by 21
  11. 2023Exploring Question Decomposition for Zero-Shot VQAcited by 12
  12. 2024Beyond Math: Stories as a Testbed for Memorization-Constrained Reasoning in LLMscited by 12
25 more
  1. 2026Adaptive Information Control for Search-Augmented LLM Reasoningcited by 11
  2. 2025Read Before You Think: Mitigating LLM Comprehension Failures with Step-by-Step Readingcited by 10
  3. 2025DRF: LLM-AGENT Dynamic Reputation Filtering Frameworkcited by 10
  4. 2025DSPC: Dual-Stage Progressive Compression Framework for Efficient Long-Context Reasoningcited by 10
  5. 2026How Early Is Early Enough? Design-Dependent Observation-Window Sufficiency in Subscription Churn Predictioncited by 8
  6. 2026From Faulty Memories to Corrected Actions: Dependency-Guided Rollback Repair for Memory-Augmented Agentscited by 8
  7. 2025Adaptive Context Length Optimization with Low-Frequency Truncation for Multi-Agent Reinforcement Learningcited by 7
  8. 2026MAVEN-T: Reinforced Heterogeneous Distillation for Real-Time Multi-Agent Trajectory Predictioncited by 4
  9. 2026Interpretable Factor Decomposition for Decision Intelligence in Large-Scale Financial Markets: Evidence from China's A-Share Marketcited by 4
  10. 2024Image First or Text First? Optimising the Sequencing of Modalities in Large Language Model Prompting and Reasoning Taskscited by 3
  11. 2025Perceptual Taxonomy: Evaluating and Guiding Hierarchical Scene Reasoning in Vision-Language Modelscited by 3
  12. 2026Disentangling Perception and Reasoning in Multimodal LLMs via Reward Designcited by 3
  13. 2026DISSECT: Diagnosing Where Vision Ends and Language Priors Begin in Scientific VLMscited by 3
  14. 2026Sheet As Token: A Graph-Enhanced Representation for Multi-Sheet Spreadsheet Understandingcited by 3
  15. 2026Validation-Stage Combinatorial Fusion Analysis for Imbalanced Credit-Card Fraud Detectioncited by 3
  16. 2026Interpretable vs Learned Encoders for High-Cardinality Fraud Detectioncited by 3
  17. 2026OSEF: One-Step Evidence Fusion for Cross-Video Scene Procedure Planningcited by 3
  18. 2026Same Question, Different Answer? Measuring and Mitigating Prompt Privilege for Equitable AI Accesscited by 3
  19. 2025Bridging Dual Knowledge Graphs for Multi-Hop Question Answering in Construction Safetycited by 2
  20. 2026Decomposing Firm-Level Crisis Responses from Incomplete Market Signals: Evidence from China's IT Sector During COVID-19cited by 2
  21. 2026Explainable Machine Learning for Broadband Adoption Disparities: Tract-Level Prediction and SHAP-Based Factor Profilingcited by 1
  22. 2026RACER: Reinforced Agent Collaboration for Explainable Reasoning on Knowledge Graphscited by 1
  23. 2026Support Thresholds, Not Algorithms, Limit Rare-Association Recovery in Co-Purchase Networkscited by 1
  24. 2026IndustrialVLA-Bench: A Traceable Multi-Axis Evaluation of Open Robot Policy Modelscited by 1
  25. 2026A Benchmark and Diagnostic Study of Epistemic Admission in Shared Agent Memorycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022When and why vision-language models behave like bags-of-words, and what to do about it?
  2. 2026Specializing Large Models for Oracle Bone Script Interpretation via Component-Grounded Multimodal Knowledge Augmentation
  3. 2026Chain-of-Procedure: Hierarchical Visual-Language Reasoning for Procedural QA
  4. 2024Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences
  5. 2025Visual-TableQA: Open-Domain Benchmark for Reasoning over Table Images
  6. 2026ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge
  7. 2026Which Source Wins? Task-Dependent Reliance in Vision-Language Models
  8. 2026More Images, More Problems? A Controlled Analysis of VLM Failure Modes