mlsanityData to 14 Oct 2026

Semantic Saliency from Multi-Modal Large Language Model Scene Understanding Maps

arXiv 2610.15467 ↗First posted 11 Oct 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning also fits: Image editing and quality assessment

Abstract (arXiv)

Image-computable low-level image saliency has shaped perceptual psychology and computer science, but is limited by its inability to capture human high-level cognitive factors that influence attention and eye movements. Here, we propose a new fully automated method to compute semantic saliency by quantifying each object's contribution to scene comprehension without fitting any parameters to human data: the Automated Scene Understanding Map (AUTO-SUM). AUTO-SUM automatically segments and removes individual objects from a scene, uses multimodal large language models (MLLMs) to describe the original and modified scenes, and measures the semantic similarity between these descriptions using text embeddings. We show that AUTO-SUM can predict which object is most important in a human-measured scene understanding map (H-SUM) estimated using human scene descriptions. We also show that AUTO-SUM can predict the object most fixated by human observers during free viewing and scene description tasks better than a low-level saliency model (GBVS, Itti-Koch), a saliency model trained based on human meaning ratings (Meaning Map), and comparable to DeepGaze, which is trained on human fixation data. We contend that AUTO-SUM can be used as a semantic saliency model that complements lower-level saliency models. Code are available at https://github.com/starsky77/AUTO-SUM-Source-Code .

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 61 references that are in the map. First posted on arXiv 11 Oct 2026.

Builds on

19 of 61 references.

References · 19

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Segment Anythingcited by 8,234
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  5. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  6. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  7. 2023Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Modelscited by 1,062
  8. 2025SAM 3: Segment Anything with Conceptscited by 906
  9. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  10. 2018Object Hallucination in Image Captioningcited by 669
  11. 2023Active Retrieval Augmented Generationcited by 516
  12. 2022MAT: Mask-Aware Transformer for Large Hole Image Inpaintingcited by 218
7 more
  1. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 169
  2. 2024Jasper and Stella: distillation of SOTA embedding modelscited by 112
  3. 2017SalGAN: Visual Saliency Prediction with Generative Adversarial Networkscited by 65
  4. 2014Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNetcited by 60
  5. 2021DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modelingcited by 56
  6. 2015DeepFix: A Fully Convolutional Neural Network for predicting Human Eye Fixationscited by 48
  7. 2018Saliency Prediction in the Deep Learning Era: Successes, Limitations, and Future Challengescited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Revealing the Gap in Human and VLM Scene Perception through Counterfactual Semantic Saliency
  2. 2022Saliency Map Verbalization: Comparing Feature Importance Representations from Model-free and Instruction-based Methods
  3. 2025Predicting Reaction Time to Comprehend Scenes with Foveated Scene Understanding Maps
  4. 2025Vision-Language Models Can't See the Obvious
  5. 2026Seeing the Scene Matters: Revealing Forgetting in Video Understanding Models with a Scene-Aware Long-Video Benchmark
  6. 2026Saliency-R1: Enforcing Interpretable and Faithful Vision-language Reasoning via Saliency-map Alignment Reward
  7. 2026Learning to See What You Need: Gaze Attention for Multimodal Large Language Models
  8. 2026The Limits of Learning from Pictures and Text: Vision-Language Models and Embodied Scene Understanding