mlsanityData to 14 Oct 2026

Hub-Spectral Activation of Latent Multimodal Knowledge

arXiv 2609.17094 ↗First posted 15 Sep 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Multimodal representation learning seeks shared representations for cross-modal retrieval and knowledge transfer. Hub-based binding reduces pairwise supervision costs, but separate hub connections cannot guarantee reliable alignment between modalities without direct joint training. We introduce Hub-Spectral Activation (HSA), a closed-form method for recovering and activating the hub-readable component of latent multimodal knowledge in frozen representations. We formalize this knowledge as source-induced cross-modal dependence and characterize the component determined by the second-order statistics of two trained hub edges. Under a second-order source model, we establish conditions for exact recovery of the complete source-induced relation and bound the dimension of its hub-readable component by the hub covariance rank. HSA composes and standardizes hub-edge statistics, extracts paired spectral directions, and combines reliability-weighted matching evidence with source-gated candidate resolution for bidirectional retrieval and prototype classification. HSA requires no target-pair supervision, gradient optimization, or backbone updates. Across 19 retrieval and 11 prototype-classification relations on ImageBind and LanguageBind, HSA raises mean bidirectional Recall@10 from 18.27% to 31.15% and mean macro Top-1 accuracy from 29.01% to 52.43%, respectively. Controlled analyses further identify valid hub-edge correspondence and leading spectral directions as key sources of retrieval gains, demonstrating the utility of latent multimodal knowledge beyond native similarity scores. Code and models are publicly available at https://github.com/Luo1Yan/HSA.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 70 of its 103 references that are in the map. First posted on arXiv 15 Sep 2026.

Builds on

70 of 103 references.

References · 70

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  5. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  7. 2014Unsupervised Domain Adaptation by Backpropagationcited by 1,917
  8. 2019Invariant Risk Minimizationcited by 1,854
  9. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  10. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  11. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  12. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
38 more
  1. 2019Contrastive Multiview Codingcited by 1,301
  2. 2016Deep CORAL: Correlation Alignment for Deep Domain Adaptationcited by 1,200
  3. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  4. 2020In Search of Lost Domain Generalizationcited by 1,054
  5. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,013
  6. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  7. 2020Out-of-Distribution Generalization via Risk Extrapolation (REx)cited by 742
  8. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  9. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  10. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  11. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  12. 2019NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understandingcited by 533
  13. 2017Look, Listen and Learncited by 458
  14. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  15. 2022Multimodal Learning with Transformers: A Surveycited by 400
  16. 2023LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignmentcited by 370
  17. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 339
  18. 2018Insights on representational similarity in neural networks with canonical correlationcited by 324
  19. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  20. 2022ULIP: Learning a Unified Representation of Language, Images, and Point Clouds for 3D Understandingcited by 267
  21. 2023Multimodal Deep Learningcited by 258
  22. 2014Understanding image representations by measuring their equivariance and equivalencecited by 243
  23. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 220
  24. 2021Revisiting Model Stitching to Compare Neural Representationscited by 209
  25. 2022Relative representations enable zero-shot latent space communicationcited by 191
  26. 2016On Deep Multi-View Representation Learning: Objectives and Optimizationcited by 162
  27. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 131
  28. 2021Generalized Shape Metrics on Neural Representationscited by 112
  29. 2024IMAGDressing-v1: Customizable Virtual Dressingcited by 100
  30. 2021Robust Audio-Visual Instance Discriminationcited by 80
  31. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  32. 2023Latent Space Translation via Semantic Alignmentcited by 52
  33. 2024UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them Allcited by 49
  34. 2022ASIF: Coupled Data Turns Unimodal Models to Multimodal Without Trainingcited by 47
  35. 2021There is More than Meets the Eye: Self-Supervised Multi-Object Detection and Tracking with Sound by Distilling Multimodal Knowledgecited by 44
  36. 2024Do Vision and Language Encoders Represent the World Similarly?cited by 44
  37. 2024OmniBind: Large-scale Omni Multimodal Representation via Binding Spacescited by 39
  38. 2023Connecting Multi-modal Contrastive Representationscited by 36

50 most cited of 70 All 70 (JSON)

Built on by

Not cited yet.