mlsanityData to 14 Oct 2026

GradientCoin: A Peer-to-Peer Decentralized Large Language Models

arXiv 2308.10502 ↗First posted 21 Aug 2023cs.LG cs.CL stat.MLcited by 23

Not on the map. Why?

Abstract (arXiv)

Since 2008, after the proposal of a Bitcoin electronic cash system, Bitcoin has fundamentally changed the economic system over the last decade. Since 2022, large language models (LLMs) such as GPT have outperformed humans in many real-life tasks. However, these large language models have several practical issues. For example, the model is centralized and controlled by a specific unit. One weakness is that if that unit decides to shut down the model, it cannot be used anymore. The second weakness is the lack of guaranteed discrepancy behind this model, as certain dishonest units may design their own models and feed them unhealthy training data. In this work, we propose a purely theoretical design of a decentralized LLM that operates similarly to a Bitcoin cash system. However, implementing such a system might encounter various practical difficulties. Furthermore, this new system is unlikely to perform better than the standard Bitcoin system in economics. Therefore, the motivation for designing such a system is limited. It is likely that only two types of people would be interested in setting up a practical system for it: bullet Those who prefer to use a decentralized ChatGPT-like software. bullet Those who believe that the purpose of carbon-based life is to create silicon-based life, such as Optimus Prime in Transformers. The reason the second type of people may be interested is that it is possible that one day an AI system like this will awaken and become the next level of intelligence on this planet.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 120 of its 177 references that are in the map. First posted on arXiv 21 Aug 2023.

Builds on

120 of 177 references.

References · 120

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022Training language models to follow instructions with human feedbackcited by 15,971
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  9. 2020Scaling Laws for Neural Language Modelscited by 6,586
  10. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  11. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
38 more
  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  2. 2019Advances and Open Problems in Federated Learningcited by 3,046
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  7. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  8. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  9. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  10. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  11. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  13. 2018Federated Learning with Non-IID Datacited by 1,108
  14. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
  15. 2019Towards Federated Learning at Scale: System Designcited by 832
  16. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  17. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  18. 2018How To Backdoor Federated Learningcited by 765
  19. 2018Annotation Artifacts in Natural Language Inference Datacited by 671
  20. 2020The Future of Digital Health with Federated Learningcited by 633
  21. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  22. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
  23. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 564
  24. 2020Federated Learning with Matched Averagingcited by 564
  25. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  26. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  27. 2016Cross-stitch Networks for Multi-task Learningcited by 467
  28. 2017Differentially Private Federated Learning: A Client Level Perspectivecited by 459
  29. 2018Adaptive Federated Learning in Resource Constrained Edge Computing Systemscited by 409
  30. 2019Federated Learning in Mobile Edge Networks: A Comprehensive Surveycited by 408
  31. 2018Split learning for health: Distributed deep learning without sharing raw patient datacited by 395
  32. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 350
  33. 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 305
  34. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 294
  35. 2018Distributed learning of deep neural network over multiple agentscited by 279
  36. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 278
  37. 2019Tighter Theory for Local SGD on Identical and Heterogeneous Datacited by 266
  38. 2019A Joint Learning and Communications Framework for Federated Learning over Wireless Networkscited by 263

50 most cited of 120 All 120 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 10; 2024: 6; 2025: 4; 2026: 3.

23 papers.

Sort:

By area:

  1. 2023How to Capture Higher-order Correlations? Generalizing Matrix Softmax Attention to Kronecker Computationcited by 44
  2. 2024Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformerscited by 32
  3. 2023A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Timecited by 30
  4. 2024The Fine-Grained Complexity of Gradient Computation for Training Large Language Modelscited by 30
  5. 2023How to Protect Copyright Data in Optimization of Large Language Models?cited by 26
  6. 2024The Computational Limits of State-Space Models and Mamba via the Lens of Circuit Complexitycited by 21
  7. 2023Why Softmax Attention Outperforms Linear Attentioncited by 15
  8. 2025High-Order Matching for One-Step Shortcut Diffusion Modelscited by 15
  9. 2023A Unified Scheme of ResNet and Softmaxcited by 12
  10. 2023Unmasking Transformers: A Theoretical Approach to Data Recovery via Attention Weightscited by 12
  11. 2023An Automatic Learning Rate Schedule Algorithm for Achieving Faster Convergence and Steeper Descentcited by 9
  12. 2023Local Convergence of Approximate Newton Method for Two Layer Nonlinear Regressioncited by 9
11 more
  1. 2026Learning Decentralized LLM Collaboration with Multi-Agent Actor Criticcited by 8
  2. 2023The Expressibility of Polynomial based Attention Schemecited by 7
  3. 2025Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretrainingcited by 7
  4. 2023A Theoretical Insight into Attack and Defense of Gradient Leakage in Transformercited by 6
  5. 2024Is Your AI Truly Yours? Leveraging Blockchain for Copyrights, Provenance, and Lineagecited by 4
  6. 2025Scaling Law Phenomena Across Regression Paradigms: Multiple and Kernel Approachescited by 4
  7. 2024Inverting the Leverage Score Gradient: An Efficient Approximate Newton Methodcited by 3
  8. 2024Blockchain Meets LLMs: A Living Survey on Bidirectional Integrationcited by 2
  9. 2025Time and Memory Trade-off of KV-Cache Compression in Tensor Transformer Decodingcited by 2
  10. 2026Trust-Aware Routing for Distributed Generative AI Inference at the Edgecited by 0
  11. 2026DeRelayL: Sustainable Decentralized Relay Learningcited by 0