mlsanityData to 14 Oct 2026

Semantic-based Distributed Learning for Diverse and Discriminative Representations

arXiv 2604.18237 ↗First posted 20 Apr 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

In large-scale distributed scenarios, increasingly complex tasks demand more intelligent collaboration across networks, requiring the joint extraction of structural representations from data samples. However, conventional task-specific approaches often result in nonstructural embeddings, leading to collapsed variability among data samples within the same class, particularly in classification tasks. To address this issue and fully leverage the intrinsic structure of data for downstream applications, we propose a novel distributed learning framework that ensures both diverse and discriminative representations. For independent and identically distributed (i.i.d.) data, we reformulate and decouple the global optimization function by introducing constraints on representation variance. The update rules are then derived and simplified using a primal-dual approach. For non-i.i.d. data distributions, we tackle the problem by clustering and virtually replicating nodes, allowing model updates within each cluster using block coordinate descent. In both cases, the resulting optimal solutions are theoretically proven to maintain discriminative and diverse properties, with a guaranteed convergence for i.i.d. conditions. Additionally, semantic information from representations is shared among nodes, reducing the need for common neural network architectures. Finally, extensive simulations on MNIST, CIFAR-10 and CIFAR-100 confirm the effectiveness of the proposed algorithms in capturing global structural representations.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 53 references that are in the map. First posted on arXiv 20 Apr 2026.

Builds on

33 of 53 references.

References · 33

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  3. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  4. 2017Federated Multi-Task Learningcited by 672
  5. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 541
  6. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 502
  7. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
  8. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
  9. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  10. 2018Distributed learning of deep neural network over multiple agentscited by 279
  11. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 227
  12. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 222
21 more
  1. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  2. 2020Learning Diverse and Discriminative Representations via the Principle of Maximal Coding Rate Reductioncited by 142
  3. 2021Distributed Learning in Wireless Networks: Recent Progress and Future Challengescited by 129
  4. 2020One-Bit Over-the-Air Aggregation for Communication-Efficient Federated Edge Learning: Design and Convergence Analysiscited by 112
  5. 2016A Survey of Multi-View Representation Learningcited by 107
  6. 2018Distributed Stochastic Gradient Tracking Methodscited by 106
  7. 2020Federated Unsupervised Representation Learningcited by 93
  8. 2021Collaborative Unsupervised Visual Representation Learning from Decentralized Datacited by 69
  9. 2022On the Principles of Parsimony and Self-Consistency for the Emergence of Intelligencecited by 50
  10. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
  11. 2022Orchestra: Unsupervised Federated Learning via Globally Consistent Clusteringcited by 45
  12. 2018Distributed Variational Representation Learningcited by 32
  13. 2021FedHM: Efficient Federated Learning for Heterogeneous Models via Low-rank Factorizationcited by 31
  14. 2013Iteration Complexity Analysis of Block Coordinate Descent Methodscited by 17
  15. 2024Rethinking the Representation in Federated Unsupervised Learning with Non-IID Datacited by 13
  16. 2024A Complexity-Based Theory of Compositionalitycited by 13
  17. 2023Distributed Learning over Networks with Graph-Attention-Based Personalizationcited by 6
  18. 2020Distributed ADMM with Synergetic Communication and Computationcited by 3
  19. 2025SheafAlign: A Sheaf-theoretic Framework for Decentralized Multimodal Alignmentcited by 2
  20. 2025Communication-Efficient Personalized Distributed Learning with Data and Node Heterogeneitycited by 1
  21. 2025Compositional Distributed Learning for Multi-View Perception: A Maximal Coding Rate Reduction Perspectivecited by 1

Built on by

Not cited yet.