mlsanityData to 14 Oct 2026

BEFORE THE FLIP: Measuring Hidden Score Shifts In Quantized Vision Language Models Before The Answer Changes for Visual Question Answering

arXiv 2609.06922 ↗First posted 7 Sep 2026cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Quantization makes vision language models (VLMs) cheaper to store and run by using fewer bits to represent their weights. While unchanged answers on visual question answering (VQA) after compression are an expected behavior, they can still hide changes in the underlying scores (log probabilities). For example, a model may still answer yes after compression, even as the score gap between yes and no shrinks. We introduce BEFORE THE FLIP to measure these hidden changes. Our method compares the score change caused by compression with the change caused by replacing the image's internal representations, or image tokens, with one fixed average token. We then increase the precision of one weight group at a time to identify where extra bits help, and test whether choosing different groups for each question offers benefits beyond shuffled controls. Among 8,277 LLaVA questions where image token replacement measurably affects the scores, 4-bit compression shifts the yes or no score gap farther toward the replacement output than 8-bit compression. Qwen shows the same pattern, but with a smaller difference. Yet only 265 of 9,000 LLaVA answers change at 4 bits. In a separate study of 1,024 calibration questions, choosing weight groups separately for each question does not outperform both shuffled controls at any tested storage budget. These findings show that compression can alter the scores behind unchanged answers, but do not establish a reliable benefit from adjusting precision for each question.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 58 of its 74 references that are in the map. First posted on arXiv 7 Sep 2026.

Builds on

58 of 74 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025LUQ: Layerwise Ultra-Low Bit Quantization for Multimodal Large Language Models28 Sep
  2. 2025MQuant: Unleashing the Inference Potential of Multimodal Large Language Models via Full Static Quantization1 Feb
  3. 2024MBQ: Modality-Balanced Quantization for Large Vision-Language Models27 Dec
  4. 2024Q-VLM: Post-training Quantization for Large Vision-Language Models10 Oct

Other references · 54

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  7. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  9. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  10. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  11. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  12. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
38 more
  1. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  2. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  3. 2018Object Hallucination in Image Captioningcited by 669
  4. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  5. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 478
  6. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  7. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  8. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  9. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 350
  10. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 339
  11. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  12. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 243
  13. 2017Predict Responsibly: Improving Fairness and Accuracy by Learning to Defercited by 233
  14. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 208
  15. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decodingcited by 174
  16. 2019Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Groundedcited by 128
  17. 2026SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Modelscited by 21
  18. 2024Towards Robust Evaluation of Unlearning in LLMs via Data Transformationscited by 20
  19. 2019Optic-Net: A Novel Convolutional Neural Network for Diagnosis of Retinal Diseases from Optical Tomography Imagescited by 15
  20. 2025Side Effects of Erasing Concepts from Diffusion Modelscited by 13
  21. 2026Bridging Reasoning Trajectories in On-Policy Distillation via Near-Future Guidancecited by 13
  22. 2024Beyond Math: Stories as a Testbed for Memorization-Constrained Reasoning in LLMscited by 12
  23. 2026Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillationcited by 11
  24. 2025FedMentalCare: Towards Privacy-Preserving Fine-Tuned LLMs to Analyze Mental Health Status Using Federated Learning Frameworkcited by 8
  25. 2026Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarkscited by 7
  26. 2024Improving Shift Invariance in Convolutional Neural Networks with Translation Invariant Polyphase Samplingcited by 6
  27. 2026Omni-Modal Dissonance Benchmark: Systematically Breaking Modality Consensus to Probe Robustness and Calibrated Abstentioncited by 6
  28. 2026TRIAGE: Evaluating Prospective Metacognitive Control in LLMs under Resource Constraintscited by 6
  29. 2025FilterRAG: Zero-Shot Informed Retrieval-Augmented Generation to Mitigate Hallucinations in VQAcited by 5
  30. 2025If We May De-Presuppose: Robustly Verifying Claims through Presupposition-Free Question Decompositioncited by 5
  31. 2025FedMentor: Domain-Aware Differential Privacy for Heterogeneous Federated LLMs in Mental Healthcited by 5
  32. 2026Cross-Lingual Sentiment Misalignment: Auditing Multilingual Language Models for Inversion Risk, Dialectal Representation, and Affective Stabilitycited by 5
  33. 2026MASQuant: Modality-Aware Smoothing Quantization for Multimodal Large Language Modelscited by 5
  34. 2025VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analysiscited by 4
  35. 2026UAT-LITE: Inference-Time Uncertainty-Aware Attention for Pretrained Transformerscited by 4
  36. 2026PA3: Policy-Aware Agent Alignment through Chain-of-Thoughtcited by 4
  37. 2026Many Dialects, Many Languages, One Cultural Lens: Evaluating Multilingual VLMs for Bengali Culture Understanding Across Historically Linked Languages and Regional Dialectscited by 3
  38. 2026AgentCollabBench: Diagnosing When Good Agents Make Bad Collaboratorscited by 3

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026New Evidence, Same Choice: Testing Physical Experiment Selection in Vision Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Token Reduction for Large Vision-Language Models
  2. 2022Compressing And Debiasing Vision-Language Pre-Trained Models for Visual Question Answering
  3. 2026Query Independent Variable Rate Visual Token Coding
  4. 2026Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark
  5. 2026Evaluating the Impact of Post-Training Quantization on Reliable VQA with Multimodal LLMs
  6. 2025QG-VTC: Question-Guided Visual Token Compression in MLLMs for Efficient VQA
  7. 2024TUBench: Benchmarking Large Vision-Language Models on Trustworthiness with Unanswerable Questions
  8. 2026Towards Understanding Best Practices for Quantization of Vision-Language Models