mlsanityData to 14 Oct 2026

Online Federation For Mixtures of Proprietary Agents with Black-Box Encoders

arXiv 2505.00216 ↗First posted 30 Apr 2025cs.LG cs.AI cs.GTcited by 1

Not on the map. Why?

Abstract (arXiv)

Most industry-standard generative AIs and feature encoders are proprietary, offering only black-box access: their outputs are observable, but their internal parameters and architectures remain hidden from the end-user. This black-box access is especially limiting when constructing mixture-of-expert type ensemble models since the user cannot optimize each proprietary AI's internal parameters. Our problem naturally lends itself to a non-competitive game-theoretic lens where each proprietary AI (agent) is inherently competing against the other AI agents, with this competition arising naturally due to their obliviousness of the AI's to their internal structure. In contrast, the user acts as a central planner trying to synchronize the ensemble of competing AIs. We show the existence of the unique Nash equilibrium in the online setting, which we even compute in closed-form by eliciting a feedback mechanism between any given time series and the sequence generated by each (proprietary) AI agent. Our solution is implemented as a decentralized, federated-learning algorithm in which each agent optimizes their structure locally on their machine without ever releasing any internal structure to the others. We obtain refined expressions for pre-trained models such as transformers, random feature models, and echo-state networks. Our ``proprietary federated learning'' algorithm is implemented on a range of real-world and synthetic time-series benchmarks. It achieves orders-of-magnitude improvements in predictive accuracy over natural benchmarks, of which there are surprisingly few due to this natural problem still being largely unexplored.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 31 of its 50 references that are in the map. First posted on arXiv 30 Apr 2025.

Builds on

31 of 50 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  6. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  7. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  8. 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,188
  9. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  10. 2018Federated Learning with Non-IID Datacited by 1,108
  11. 2019Federated Learning with Differential Privacy: Algorithms and Performance Analysiscited by 536
  12. 2021SLIP: Self-supervision meets Language-Image Pre-trainingcited by 436
19 more
  1. 2019The generalization error of random features regression: Precise asymptotics and double descent curvecited by 413
  2. 2021User-friendly introduction to PAC-Bayes boundscited by 239
  3. 2018Echo state networks are universalcited by 87
  4. 2022I Know What You Trained Last Summer: A Survey on Stealing Machine Learning Models and Defencescited by 75
  5. 2020Approximation Bounds for Random Neural Networks and Reservoir Systemscited by 52
  6. 2021Optimality and Stability in Federated Learning: A Game-theoretic Approachcited by 40
  7. 2021One for One, or All for All: Equilibria and Optimality of Collaboration in Federated Learningcited by 39
  8. 2022Random Feature Amplification: Feature Learning and Generalization in Neural Networkscited by 35
  9. 2020Achieving Security and Privacy in Federated Learning Systems: Survey, Research Challenges and Future Directionscited by 25
  10. 2022Random Fully Connected Neural Networks as Perturbatively Solvable Hierarchiescited by 24
  11. 2022Integrating Random Effects in Deep Neural Networkscited by 22
  12. 2022A General Theory for Federated Optimization with Asynchronous and Heterogeneous Clients Updatescited by 17
  13. 2021The Separation Capacity of Random Neural Networkscited by 14
  14. 2024Characterizing Overfitting in Kernel Ridgeless Regression Through the Eigenspectrumcited by 13
  15. 2023Injectivity of ReLU networks: perspectives from statistical physicscited by 10
  16. 2022Collaboration in Participant-Centric Federated Learning: A Game-Theoretical Perspectivecited by 9
  17. 2022Universality and approximation bounds for echo state networks with random weightscited by 5
  18. 2025Multiplayer Federated Learning: Reaching Equilibrium with Less Communicationcited by 5
  19. 2024Random ReLU Neural Networks as Non-Gaussian Processescited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026MEMOA: Massive Mixtures of Online Agents via Mean-Field Decentralized Nash Equilibriacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Expert Merging in Sparse Mixture of Experts with Nash Bargaining
  2. 2026Mixture-of-Models: Unifying Heterogeneous Agents via N-Way Self-Evaluating Deliberation
  3. 2025MoB: Mixture of Bidders
  4. 2025A Theoretical Framework for Auxiliary-Loss-Free Load Balancing of Sparse Mixture-of-Experts in Large-Scale AI Models
  5. 2024FedMoE-DA: Federated Mixture of Experts via Domain Aware Fine-grained Aggregation
  6. 2025Online Mixture of Experts: No-Regret Learning for Optimal Collective Decision-Making
  7. 2026Matching Multiple Experts: On the Exploitability of Multi-Agent Imitation Learning
  8. 2023Federated Ensemble-Directed Offline Reinforcement Learning