mlsanityData to 7 Oct 2026

Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Models

arXiv 2606.04385 ↗First posted 3 Jun 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Foundation models have driven rapid progress in computer vision, yet the two dominant paradigms, vision-language foundation models (VLMs) and vision-only foundation models (VFMs), remain only partially compatible. VLMs offer language-grounded semantic alignment but are often visually coarse, while VFMs learn discriminative perceptual geometry but lack semantic grounding. We propose GPUA (Geometry-Preserving Unsupervised Alignment), a framework that integrates the complementary strengths of VFMs and VLMs. Inspired by cross-lingual alignment, GPUA treats VFM features as a visual language and learns an orthogonal mapping that translates the VFM space into the VLM semantic space, preserving geometry and narrowing the modality gap without labels or model parameter updates. GPUA is task-agnostic and requires only feature-level access to pretrained models. Experiments across diverse benchmarks demonstrate improved cross-model compatibility and strong gains in downstream zero-shot recognition and segmentation with negligible overhead. Code is available at https://github.com/Yuteam14/GPUA

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 54 references that are in the map. First posted on arXiv 3 Jun 2026.

Builds on

40 of 54 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025COSMIC: Clique-Oriented Semantic Multi-space Integration for Robust CLIP Test-Time Adaptation30 Mar
  2. 2025Realistic Test-Time Adaptation of Vision-Language Models7 Jan
  3. 2024Self-Calibrated CLIP for Training-Free Open-Vocabulary Segmentation24 Nov
  4. 2024Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models16 Oct
  5. 2024Frustratingly Easy Test-Time Adaptation of Vision-Language Models28 May
  6. 2024On the test-time zero-shot generalization of vision-language models: Do we really need prompt learning?3 May
  7. 2024Label Propagation for Zero-shot Classification with Vision-Language Models5 Apr
  8. 2024Efficient Test-Time Adaptation of Vision-Language Models27 Mar
  9. 2024Dual Memory Networks: A Versatile Adaptation Approach for Vision-Language Models26 Mar
  10. 2023Black Box Few-Shot Adaptation for Vision-Language models4 Apr

Other references · 30

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  3. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  5. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  6. 2013Describing Textures in the Wildcited by 2,110
  7. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  8. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  9. 2025DINOv3cited by 1,297
  10. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  11. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  12. 2017Word Translation Without Parallel Datacited by 480
18 more
  1. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 357
  2. 2013Exploiting Similarities among Languages for Machine Translationcited by 187
  3. 2022MaskCLIP: Masked Self-Distillation Advances Contrastive Language-Image Pretrainingcited by 187
  4. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
  5. 2018A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddingscited by 145
  6. 2023SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensingcited by 123
  7. 2024ProxyCLIP: Proxy Attention Improves CLIP for Open-Vocabulary Segmentationcited by 110
  8. 2018Unsupervised Alignment of Embeddings with Wasserstein Procrustescited by 96
  9. 2023CLIP-DINOiser: Teaching CLIP a few DINO tricks for open-vocabulary semantic segmentationcited by 94
  10. 2024DINOv2 Meets Text: A Unified Framework for Image- and Pixel-Level Vision-Language Alignmentcited by 79
  11. 2023CLIP as RNN: Segment Countless Visual Concepts without Training Endeavorcited by 69
  12. 2024Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentationcited by 42
  13. 2024Enhance Vision-Language Alignment with Noisecited by 34
  14. 2024CorrCLIP: Reconstructing Patch Correlations in CLIP for Open-Vocabulary Semantic Segmentationcited by 33
  15. 2018Learning Multilingual Word Embeddings in Latent Metric Space: A Geometric Approachcited by 22
  16. 2025LPOSS: Label Propagation Over Patches and Pixels for Open-vocabulary Semantic Segmentationcited by 21
  17. 2024Tuning-free Universally-Supervised Semantic Segmentationcited by 6
  18. 2025SOTA: Self-adaptive Optimal Transport for Zero-Shot Classification with Multiple Foundation Modelscited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Modality Alignment across Trees on Heterogeneous Hyperbolic Manifolds
  2. 2026Geometric Similarity in VLM Low-Level Vision Representations
  3. 2026Topology-Aware Representation Alignment for Semi-Supervised Vision-Language Learning
  4. 2026Rank-Aware Hyperbolic Alignment for Vision-Language Dataset Distillation
  5. 2026Underrepresented in Foundation Model Pretraining Data? A One-Shot Probe
  6. 2026Supervised Classification Heads as Semantic Prototypes: Unlocking Vision-Language Alignment via Weight Recycling
  7. 2025Prompt the Unseen: Evaluating Visual-Language Alignment Beyond Supervision
  8. 2026GeoWorld-VLM: Geometry from World Models for Vision-Language Models