mlsanityData to 14 Oct 2026

Federated Cross-Modal Retrieval with Missing Modalities via Semantic Routing and Adapter Personalization

arXiv 2604.22885 ↗First posted 24 Apr 2026cs.CV cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated cross-modal retrieval faces severe challenges from heterogeneous client data, particularly non-IID semantic distributions and missing modalities. Under such heterogeneity, a single global model is often insufficient to capture both shared cross-modal knowledge and client-specific characteristics. We propose RCSR, a personalization-friendly federated framework that integrates prototype anchoring, retrieval-centric semantic routing, and optional client-specific adapters. Built on a frozen CLIP backbone, RCSR leverages lightweight shared adapters for global knowledge transfer while supporting efficient local personalization. Prototype anchoring helps unimodal clients align with global cross-modal semantics, and a server-side semantic router adaptively assigns aggregation weights based on retrieval consistency to mitigate alignment drift during heterogeneous updates. Extensive experiments on MS-COCO, Flickr30K, and other benchmarks show that RCSR consistently improves global retrieval accuracy and training stability, while further enhancing client-level retrieval performance, especially for clients with incomplete modalities. Code is available at https://github.com/RezinChow/RCSR-Retrieval-Centric-Semantic-Routing.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 41 references that are in the map. First posted on arXiv 24 Apr 2026.

Builds on

24 of 41 references.

References · 24

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  5. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  6. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  7. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  8. 2021Model-Contrastive Federated Learningcited by 766
  9. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  10. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
  11. 2020Personalized Federated Learning with Moreau Envelopescited by 618
  12. 2019Federated Learning with Personalization Layerscited by 541
12 more
  1. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  2. 2017Differentially Private Federated Learning: A Client Level Perspectivecited by 459
  3. 2019Agnostic Federated Learningcited by 456
  4. 2018Stacked Cross Attention for Image-Text Matchingcited by 416
  5. 2019Fair Resource Allocation in Federated Learningcited by 394
  6. 2019A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protectioncited by 374
  7. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
  8. 2020Personalized Federated Learning: A Meta-Learning Approachcited by 312
  9. 2015Visualizing and Understanding Recurrent Networkscited by 160
  10. 2023Multimodal Federated Learning via Contrastive Representation Ensemblecited by 86
  11. 2024Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modalitycited by 21
  12. 2023FedMEKT: Distillation-based Embedding Knowledge Transfer for Multimodal Federated Learningcited by 17

Built on by

Not cited yet.