mlsanityData to 7 Oct 2026

Geometric Similarity in VLM Low-Level Vision Representations

arXiv 2610.00848 ↗First posted 1 Oct 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning also fits: Prompt learning and adaptation of CLIP

also in 1 more
Abstract (arXiv)

Vision-language models (VLMs) have emerged as powerful candidates for universal vision backbones, with representative architectures including autoregressive (AR) models and diffusion transformers (DiTs). Yet, adapting them efficiently for all-in-one low-level image restoration remains a challenge. Crucially, the field lacks an understanding of how VLMs organize hidden-layer representations and whether these structurally distinct paradigms share a common geometric organization for pixel-level perception. Such shared organization is a prerequisite for building highly transferable, unified restoration VLMs and adapters. In this paper, we systematically investigate representational similarity across 24 low-level tasks spanning 5 categories. We propose GeoSim, a unified four-level framework that analyzes task-conditioned representations from global similarity, local geometry, sparse feature decomposition, and topological verification perspectives. Our formulation applies to the analysis of hidden states in AR models and feature maps in DiTs across same- and cross-task/model settings. Our results reveal the organizing principles of low-level visual representations while exposing their limits in cross-task and cross-model agreement. Ultimately, GeoSim provides an interpretability lens for probing latent transferability in low-level vision and diagnosing model limitations in task- or model-specific scenarios.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 75 of its 103 references that are in the map. First posted on arXiv 1 Oct 2026.

Builds on

75 of 103 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Steering Vision-Language Models with Joint Sparse Autoencoders24 Jun
  2. 2026Extraction and Analysis of Multimodal Concepts in Vision Language Models through Sparse Autoencoders19 Jun
  3. 2025Understanding Task Transfer in Vision-Language Models24 Nov
  4. 2025Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models3 Apr
  5. 2024Towards Interpreting Visual Information Processing in Vision-Language Models9 Oct
  6. 2024What Are We Measuring When We Evaluate Large Vision-Language Models? An Analysis of Latent Factors and Biases3 Apr
  7. 2023Vision-Language Models for Vision Tasks: A Survey3 Apr

Other references · 68

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  4. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  5. 2025Qwen3-VL Technical Reportcited by 2,534
  6. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  7. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  8. 2021Grounded Language-Image Pre-trainingcited by 1,147
  9. 2025Qwen-Image Technical Reportcited by 973
  10. 2025Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scalingcited by 787
  11. 2024Emu3: Next-Token Prediction is All You Needcited by 761
  12. 2016Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurringcited by 722
38 more
  1. 2018Deep Retinex Decomposition for Low-Light Enhancementcited by 720
  2. 2022Language-driven Semantic Segmentationcited by 711
  3. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  4. 2018Taskonomy: Disentangling Task Transfer Learningcited by 631
  5. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  6. 2024Scaling and evaluating sparse autoencoderscited by 530
  7. 2019Which Tasks Should Be Learned Together in Multi-task Learning?cited by 367
  8. 2019An Underwater Image Enhancement Benchmark Dataset and Beyondcited by 250
  9. 2021TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditionscited by 243
  10. 2017DesnowNet: Context-Aware Deep Network for Snow Removalcited by 200
  11. 2019Task2Vec: Task Embedding for Meta-Learningcited by 197
  12. 2026Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnetcited by 175
  13. 2017Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removalcited by 160
  14. 2024InstructIR: High-Quality Image Restoration Following Human Instructionscited by 139
  15. 2020Defocus Deblurring Using Dual-Pixel Datacited by 138
  16. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inferencecited by 137
  17. 2019DoveNet: Deep Image Harmonization via Domain Verificationcited by 134
  18. 2025Emu3.5: Native Multimodal Models are World Learnerscited by 119
  19. 2024Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)cited by 116
  20. 2024ANOLE: An Open, Autoregressive, Native Large Multimodal Models for Interleaved Image-Text Generationcited by 112
  21. 2021Generalized Shape Metrics on Neural Representationscited by 109
  22. 2023Controlling Vision-Language Models for Multi-Task Image Restorationcited by 108
  23. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinationscited by 103
  24. 2023AutoDIR: Automatic All-in-One Image Restoration with Latent Diffusioncited by 98
  25. 2019Representation Similarity Analysis for Efficient Task taxonomy & Transfer Learningcited by 86
  26. 2021Open-Vocabulary Instance Segmentation via Robust Cross-Modal Pseudo-Labelingcited by 84
  27. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Modelscited by 73
  28. 2024Sparse autoencoders reveal selective remapping of visual concepts during adaptationcited by 66
  29. 2025Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignmentcited by 62
  30. 2024A Survey on All-in-One Image Restoration: Taxonomy, Evaluation and Future Trendscited by 58
  31. 2025Interpreting CLIP with Hierarchical Sparse Autoencoderscited by 54
  32. 2025Harnessing the Universal Geometry of Embeddingscited by 54
  33. 2023Multimodal Prompt Perceiver: Empower Adaptiveness, Generalizability and Fidelity for All-in-One Image Restorationcited by 50
  34. 2018Effects of Degradations on Deep Neural Network Architecturescited by 47
  35. 2026When Models Manipulate Manifolds: The Geometry of a Counting Taskcited by 43
  36. 2024RestoreAgent: Autonomous Image Restoration Agent via Multimodal Large Language Modelscited by 41
  37. 2019A Comprehensive Benchmark for Single Image Compression Artifacts Reductioncited by 31
  38. 2023Flare7K++: Mixing Synthetic and Real Datasets for Nighttime Flare Removal and Beyondcited by 31

50 most cited of 68 All 68 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance
  2. 2026Through the LENS: Local Geometric Decomposition of Vision-Language Model Representations
  3. 2026GeoWorld-VLM: Geometry from World Models for Vision-Language Models
  4. 2025PairBench: Are Vision-Language Models Reliable at Comparing What They See?
  5. 2026Why Far Looks Up: Probing Spatial Representation in Vision-Language Models
  6. 2024Locality Alignment Improves Vision-Language Models
  7. 2026Stateful Visual Encoders for Vision-Language Models
  8. 2026GeoPID: Decomposing and Steering Visual Information in Vision-Language Models