mlsanityData to 7 Oct 2026

IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representations

arXiv 2404.01266 ↗First posted 1 Apr 2024cs.AI cs.CLcited by 37

Not on the map. Why?

Cited by 23 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Current foundation models exhibit impressive capabilities when prompted either with text only or with both image and text inputs. But do their capabilities change depending on the input modality? In this work, we propose IsoBench, a benchmark dataset containing problems from four major areas: math, science, algorithms, and games. Each example is presented with multiple isomorphic representations of inputs, such as visual, textual, and mathematical presentations. IsoBench provides fine-grained feedback to diagnose performance gaps caused by the form of the representation. Across various foundation models, we observe that on the same problem, models have a consistent preference towards textual representations. Most prominently, when evaluated on all IsoBench problems, Claude-3 Opus performs 28.7 points worse when provided with images instead of text; similarly, GPT-4 Turbo is 18.7 points worse and Gemini Pro is 14.9 points worse. Finally, we present two prompting techniques, textitIsoCombination and textitIsoScratchPad, which improve model performance by considering combinations of, and translations between, different input representations.

Key facts. Cited by 37 papers in the map (data through 7 Oct 2026). Builds on 40 of its 61 references that are in the map. First posted on arXiv 1 Apr 2024.

Builds on

40 of 61 references.

References · 40

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  5. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  6. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  8. 2024Mixtral of Expertscited by 1,785
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  10. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  11. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  12. 2022Transformers learn in-context by gradient descentcited by 714
28 more
  1. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  2. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  3. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 254
  4. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  5. 2022Demystifying Prompts in Language Models via Perplexity Estimationcited by 215
  6. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  7. 2021GPT3Mix: Leveraging Large-scale Language Models for Text Augmentationcited by 185
  8. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  9. 2023Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuningcited by 168
  10. 2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
  11. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  12. 2023Teaching Arithmetic to Small Transformerscited by 121
  13. 2023NEFTune: Noisy Embeddings Improve Instruction Finetuningcited by 99
  14. 2023Quantifying & Modeling Multimodal Interactions: An Information Decomposition Frameworkcited by 81
  15. 2022CounTR: Transformer-based Generalised Visual Countingcited by 79
  16. 2023Is GPT-4 a Good Data Analyst?cited by 69
  17. 2022Simplicity Bias in Transformers and their Ability to Learn Sparse Boolean Functionscited by 68
  18. 2022Data Curation Alone Can Stabilize In-context Learningcited by 58
  19. 2023Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regressioncited by 47
  20. 2023Interpretable Diffusion via Information Decompositioncited by 37
  21. 2022ORCA: Interpreting Prompted Language Models via Locating Supporting Data Evidence in the Ocean of Pretraining Datacited by 34
  22. 2018Shifting the Baseline: Single Modality Performance on Visual Navigation & QAcited by 30
  23. 2024How Far Are We from Intelligent Visual Deductive Reasoning?cited by 29
  24. 2024DeLLMa: Decision Making Under Uncertainty with Large Language Modelscited by 20
  25. 2024Transformers Learn Low Sensitivity Functions: Investigations and Implicationscited by 13
  26. 2023Improved Instruction Ordering in Recipe-Grounded Conversationcited by 7
  27. 2024Towards Understanding the Word Sensitivity of Attention Layers: A Study via Random Featurescited by 5
  28. 2023Approximating CKY with Transformerscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 22; 2026: 2.

37 papers.

Sort:

By area:

  1. 2025Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmarkcited by 132
  2. 2024Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Modelscited by 111
  3. 2025Zebra-CoT: A Dataset for Interleaved Vision Language Reasoningcited by 78
  4. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  5. 2025ThinkMorph: Emergent Properties in Multimodal Interleaved Chain-of-Thought Reasoningcited by 55
  6. 2024ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detectioncited by 37
  7. 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMscited by 35
  8. 2025Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thoughtcited by 34
  9. 2025Evaluating and Steering Modality Preferences in Multimodal Large Language Modelcited by 33
  10. 2024Distill Visual Chart Reasoning Ability from LLMs to MLLMscited by 30
  11. 2024Do Vision & Language Decoders use Images and Text equally? How Self-consistent are their Explanations?cited by 23
  12. 2024Disentangling Logic: The Role of Context in Large Language Model Reasoning Capabilitiescited by 22
25 more
  1. 2025Unfolding Spatial Cognition: Evaluating Multimodal Models on Visual Simulationscited by 21
  2. 2025Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Modelscited by 16
  3. 2024TLDR: Token-Level Detective Reward Model for Large Vision Language Modelscited by 15
  4. 2025Investigating Inference-time Scaling for Chain of Multi-modal Thought: A Preliminary Studycited by 11
  5. 2024Modality-Specialized Synergizers for Interleaved Vision-Language Generalistscited by 9
  6. 2025Interactive Sketchpad: A Multimodal Tutoring System for Collaborative, Visual Problem-Solvingcited by 9
  7. 202534 Examples of LLM Applications in Materials Science and Chemistry: Towards Automation, Assistants, Agents, and Accelerated Scientific Discoverycited by 9
  8. 2024Reflections from the 2024 Large Language Model (LLM) Hackathon for Applications in Materials Science and Chemistrycited by 8
  9. 2024MetaScientist: A Human-AI Synergistic Framework for Automated Mechanical Metamaterial Designcited by 8
  10. 2025Textual Steering Vectors Can Improve Visual Understanding in Multimodal Large Language Modelscited by 8
  11. 2025ChessQA: Evaluating Large Language Models for Chess Understandingcited by 8
  12. 2025Octopus: Agentic Multimodal Reasoning with Six-Capability Orchestrationcited by 8
  13. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflictcited by 7
  14. 2025MAPS: Multi-Agent Personality Shaping for Collaborative Reasoningcited by 4
  15. 2025VISaGE: Understanding Visual Generics and Exceptionscited by 4
  16. 2026Mechanisms of Prompt-Induced Hallucination in Vision-Language Modelscited by 4
  17. 2024LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Models for Referring Expression Comprehensioncited by 3
  18. 2025PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Modelscited by 3
  19. 2025Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoningcited by 3
  20. 2025MapTrace: Scalable Data Generation for Route Tracing on Mapscited by 3
  21. 2024LocateBench: Evaluating the Locating Ability of Vision Language Modelscited by 2
  22. 2025Domain Specific Benchmarks for Evaluating Multimodal Large Language Modelscited by 2
  23. 2024VisualLens: Personalization through Task-Agnostic Visual Historycited by 0
  24. 2025Is your multimodal large language model a good science tutor?cited by 0
  25. 2026I Don't Miss You, but I Do: Self-Explanation Faithfulness of Modality Missingness in Vision-Language Modelscited by 0