mlsanityData to 14 Oct 2026

Induced Numerical Instability: Hidden Costs in Multimodal Large Language Models

arXiv 2603.04453 ↗First posted 27 Feb 2026cs.CL cs.AI cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

The use of multimodal large language models has become widespread, and as such the study of these models and their failure points has become of utmost importance. We study a novel mode of failure that causes degradation in performance indirectly by optimizing a loss term that seeks to maximize numerical instability in the inference stage of these models. We apply this loss term as the optimization target to construct images that, when used on multimodal large language models, cause significant degradation in the output. We validate our hypothesis on state of the art models large vision language models (LLaVa-v1.5-7B, Idefics3-8B, SmolVLM-2B-Instruct) against standard datasets (Flickr30k, MMVet, TextVQA, VQAv2, POPE, COCO) and show that performance degrades significantly, even with a very small change to the input image, compared to baselines. Our results uncover a fundamentally different vector of performance degradation, highlighting a failure mode not captured by adversarial perturbations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 28 of its 37 references that are in the map. First posted on arXiv 27 Feb 2026.

Builds on

28 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct

Other references · 27

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2014Explaining and Harnessing Adversarial Examplescited by 7,364
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  6. 2022Emergent Abilities of Large Language Modelscited by 2,512
  7. 2015VQA: Visual Question Answeringcited by 2,506
  8. 2016Adversarial examples in the physical worldcited by 1,769
  9. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  10. 2019Towards VQA Models That Can Readcited by 1,718
  11. 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
  12. 2017Mixed Precision Trainingcited by 1,156
15 more
  1. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 812
  2. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  3. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 415
  4. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 237
  5. 2024Building and better understanding vision-language models: insights and future directionscited by 173
  6. 2025Alita: Generalist Agent Enabling Scalable Agentic Reasoning with Minimal Predefinition and Maximal Self-Evolutioncited by 116
  7. 2025MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Serverscited by 96
  8. 2023On the Robustness of Large Multimodal Models Against Image Adversarial Attackscited by 92
  9. 2025Grokking at the Edge of Numerical Stabilitycited by 46
  10. 2025Code Researcher: Deep Research Agent for Large Systems Code and Commit Historycited by 20
  11. 2024FACTS About Building Retrieval Augmented Generation-based Chatbotscited by 15
  12. 2024On Large Language Models in Mission-Critical IT Governance: Are We Ready Yet?cited by 9
  13. 2024Intriguing Properties of Robust Classificationcited by 3
  14. 2025Safeguarding Vision-Language Models: Mitigating Vulnerabilities to Gaussian Noise in Perturbation-based Attackscited by 3
  15. 2022Beyond NaN: Resiliency of Optimization Layers in The Face of Infeasibilitycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Breach in the Shield: Unveiling the Vulnerabilities of Large Language Models
  2. 2026Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models
  3. 2025Res-Bench: Benchmarking the Robustness of Multimodal Large Language Models to Dynamic Resolution Input
  4. 2025Reasoning Multimodal Large Language Model: Data Contamination and Dynamic Evaluation
  5. 2025Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models
  6. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
  7. 2026Multimodal Large Language Models as Image Classifiers
  8. 2026Reducing Peak Memory Usage for Modern Multimodal Large Language Model Pipelines