mlsanityData to 14 Oct 2026

When More Foundation Models Means Less: Diagnosing and Addressing Multi-View Fusion Failure

arXiv 2608.17490 ↗First posted 18 Aug 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Foundation-model hubs turn multi-view fusion into a selection problem: from a large heterogeneous encoder pool, which views should be fused, and how many? We show that downstream performance is non-monotonic in the number of fused encoders; later views can be redundant or task-misaligned, causing accuracy to saturate or decline. We formalise this setting as view-set composition and propose KAGES (Kernel-Alignment Greedy Encoder Selector), a label-aware method that orders frozen encoders by their marginal gain in centred kernel-target alignment. KAGES requires no downstream classifier training during selection, evaluates each candidate in O(n2) time independent of encoder dimension, and admits a conditional (1-e-γ) prefix-wise guarantee under monotonicity and a positive submodularity ratio. Across five recognition regimes and low-shot, larger-pool, and full-data protocols, KAGES improves average AULC over full fusion by 3.9, 5.8, and 3.3 points, respectively, and exceeds DPP and facility-location selection in average AULC. Image retrieval exhibits later, task-dependent saturation along the KAGES ordering, while peak-then-decline reproduces in frozen-LLM fusion. These results show that effective large-pool fusion depends on selecting a compact, task-aligned set of views rather than indiscriminately fusing more encoders.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 39 of its 52 references that are in the map. First posted on arXiv 18 Aug 2026.

Builds on

39 of 52 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2023Segment Anythingcited by 8,234
  7. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  9. 2024Qwen2.5 Technical Reportcited by 4,942
  10. 2022A ConvNet for the 2020scited by 3,579
  11. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  12. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
27 more
  1. 2019Similarity of Neural Network Representations Revisitedcited by 1,755
  2. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
  3. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  4. 2024The Platonic Representation Hypothesiscited by 382
  5. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 362
  6. 2014On the Complexity of Best Arm Identification in Multi-Armed Bandit Modelscited by 286
  7. 2020LEEP: A New Measure to Evaluate Transferability of Learned Representationscited by 188
  8. 2021LogME: Practical Assessment of Pre-trained Models for Transfer Learningcited by 172
  9. 2013A Survey on Multi-view Learningcited by 152
  10. 2023Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learnerscited by 152
  11. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 139
  12. 2019Transferability and Hardness of Supervised Classification Taskscited by 127
  13. 2022Task Residual for Tuning Vision-Language Modelscited by 123
  14. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Modelscited by 98
  15. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Contextcited by 83
  16. 2022An Information-Theoretic Approach to Transferability in Task Transfer Learningcited by 81
  17. 2021OTCE: A Transferability Metric for Cross-Domain Cross-Task Representationscited by 62
  18. 2021Reconsidering Representation Alignment for Multi-view Clusteringcited by 57
  19. 2021Transferability Estimation using Bhattacharyya Class Separabilitycited by 54
  20. 2022PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Taskscited by 30
  21. 2021Ranking and Tuning Pre-trained Models: A New Paradigm for Exploiting Model Hubscited by 28
  22. 2023Foundation Model is Efficient Multimodal Multitask Model Selectorcited by 16
  23. 2022Transferability-Guided Cross-Domain Cross-Task Transfer Learningcited by 14
  24. 2023Explore and Exploit the Diverse Knowledge in Model Zoo for Domain Generalizationcited by 14
  25. 2022ZooD: Exploiting Model Zoo for Out-of-Distribution Generalizationcited by 12
  26. 2024Assessing Pre-Trained Models for Transfer Learning Through Distribution of Spectral Componentscited by 6
  27. 2023Building a Winning Team: Selecting Source Model Ensembles using a Submodular Transferability Estimation Approachcited by 5

Built on by

Not cited yet.