mlsanityData to 14 Oct 2026

Multi-Modal Multi-Task Federated Foundation Models for Next-Generation Extended Reality Systems: Towards Privacy-Preserving Distributed Intelligence in AR/VR/MR

arXiv 2506.05683 ↗First posted 6 Jun 2025cs.LG cs.AI cs.CRcited by 3

Not on the map. Why?

Abstract (arXiv)

Extended reality (XR) systems, which consist of virtual reality (VR), augmented reality (AR), and mixed reality (XR), offer a transformative interface for immersive, multi-modal, and embodied human-computer interaction. In this paper, we envision that multi-modal multi-task (M3T) federated foundation models (FedFMs) can offer transformative capabilities for XR systems through integrating the representational strength of M3T foundation models (FMs) with the privacy-preserving model training principles of federated learning (FL). We present a modular architecture for FedFMs, which entails different coordination paradigms for model training and aggregations. Central to our vision is the codification of XR challenges that affect the implementation of FedFMs under the SHIFT dimensions: (1) Sensor and modality diversity, (2) Hardware heterogeneity and system-level constraints, (3) Interactivity and embodied personalization, (4) Functional/task variability, and (5) Temporality and environmental variability. We illustrate the manifestation of these dimensions across a set of emerging and anticipated applications of XR systems. Finally, we propose evaluation metrics, dataset requirements, and design tradeoffs necessary for the development of resource-aware FedFMs in XR. This perspective aims to chart the technical and conceptual foundations for context-aware privacy-preserving intelligence in the next generation of XR systems.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 39 of its 86 references that are in the map. First posted on arXiv 6 Jun 2025.

Builds on

39 of 86 references.

References · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  6. 2022Visual Prompt Tuningcited by 1,681
  7. 2022Diffusion Models: A Comprehensive Survey of Methods and Applicationscited by 1,114
  8. 2023A Comprehensive Overview of Large Language Modelscited by 648
  9. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 327
  10. 2022AUC Maximization in the Era of Big Data and AI: A Surveycited by 100
  11. 2023When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directionscited by 84
  12. 2019xR-EgoPose: Egocentric 3D Human Pose from an HMD Cameracited by 79
27 more
  1. 2019Federated Learning with Cooperating Devices: A Consensus Approach for Massive IoT Networkscited by 79
  2. 2023FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learningcited by 60
  3. 2024Dual-Personalizing Adapter for Federated Foundation Modelscited by 55
  4. 2020Federated Learning in Vehicular Networkscited by 53
  5. 2021Semi-Decentralized Federated Learning with Cooperative D2D Local Model Aggregationscited by 51
  6. 2024Federated LoRA with Sparse Communicationcited by 26
  7. 2024Delving into Multi-modal Multi-task Foundation Models for Road Scene Understanding: From Learning Paradigm Perspectivescited by 23
  8. 2024Configurable Foundation Models: Building LLMs from a Modular Perspectivecited by 22
  9. 2021Federated Learning for Industrial Internet of Things in Future Industriescited by 14
  10. 2024Open Challenges and Opportunities in Federated Foundation Models Towards Biomedical Healthcarecited by 12
  11. 2025Few-Shot Adversarial Low-Rank Fine-Tuning of Vision-Language Modelscited by 9
  12. 2023A Human-Centric Metaverse Enabled by Brain-Computer Interface: A Surveycited by 7
  13. 2024EmBARDiment: an Embodied AI Agent for Productivity in XRcited by 7
  14. 2024Multimodal Object Detection using Depth and Image Data for Manufacturing Partscited by 7
  15. 2025Multi-Modal Multi-Task (M3T) Federated Foundation Models for Embodied AI: Potentials and Challenges for Edge Integrationcited by 6
  16. 2022MUDGUARD: Taming Malicious Majorities in Federated Learning using Privacy-Preserving Byzantine-Robust Clusteringcited by 5
  17. 2024Headset: Human emotion awareness under partial occlusions multimodal datasetcited by 4
  18. 2024A Comprehensive Review of Multimodal XR Applications, Risks, and Ethical Challenges in the Metaversecited by 4
  19. 2023BONES: Near-Optimal Neural-Enhanced Video Streamingcited by 3
  20. 2024IllusionX: An LLM-powered mixed reality personal companioncited by 3
  21. 2024Dynamic D2D-Assisted Federated Learning over O-RAN: Performance Analysis, MAC Scheduler, and Asymmetric User Selectioncited by 3
  22. 2025Redefining non-IID Data in Federated Learning for Computer Vision Tasks: Migrating from Labels to Embeddings for Task-Specific Data Distributionscited by 3
  23. 2025MENA: Multimodal Epistemic Network Analysis for Visualizing Competencies and Emotionscited by 3
  24. 2025A Survey on Foundation Models for Personalized Federated Intelligencecited by 3
  25. 2025Neural-Enhanced Rate Adaptation and Computation Distribution for Emerging mmWave Multi-User 3D Video Streaming Systemscited by 3
  26. 2025Towards spatial computing: recent advances in multimodal natural interaction for XR headsetscited by 2
  27. 2024Heads Up eXperience (HUX): Always-On AI Companion for Human Computer Environment Interactioncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Hierarchical Federated Foundation Models over Wireless Networks for Multi-Modal Multi-Task Intelligence: Integration of Edge Learning with D2D/P2P-Enabled Fog Learning Architecturescited by 2
  2. 2025Bringing Multi-Modal Multi-Task Federated Foundation Models to Education Domain: Prospects and Challengescited by 0
  3. 2026BLEND: Balancing Personalization vs. Generalization in Federated Vision-Language Modelscited by 0