mlsanityData to 14 Oct 2026

Trillion Parameter AI Serving Infrastructure for Scientific Discovery: A Survey and Vision

arXiv 2402.03480 ↗First posted 5 Feb 2024cs.LG cs.AI cs.DCcited by 5

Not on the map. Why?

Abstract (arXiv)

Deep learning methods are transforming research, enabling new techniques, and ultimately leading to new discoveries. As the demand for more capable AI models continues to grow, we are now entering an era of Trillion Parameter Models (TPM), or models with more than a trillion parameters -- such as Huawei's PanGu-Sigma. We describe a vision for the ecosystem of TPM users and providers that caters to the specific needs of the scientific community. We then outline the significant technical challenges and open problems in system design for serving TPMs to enable scientific research and discovery. Specifically, we describe the requirements of a comprehensive software stack and interfaces to support the diverse and flexible requirements of researchers.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 34 of its 65 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

34 of 65 references.

References · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  6. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  7. 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,576
  8. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  9. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  10. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  11. 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
  12. 2016Automatic chemical design using a data-driven continuous representation of moleculescited by 765
22 more
  1. 2022Galactica: A Large Language Model for Sciencecited by 744
  2. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  3. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  4. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  5. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  6. 2017Progressive Neural Architecture Searchcited by 487
  7. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  8. 2014Caffe: Convolutional Architecture for Fast Feature Embeddingcited by 288
  9. 2020A Comprehensive Survey of Neural Architecture Search: Challenges and Solutionscited by 254
  10. 2017graph2vec: Learning Distributed Representations of Graphscited by 230
  11. 2016Clipper: A Low-Latency Online Prediction Serving Systemcited by 191
  12. 2018Neural Architecture Optimizationcited by 167
  13. 2023PanGu-Sigma: Towards Trillion Parameter Language Model with Sparse Heterogeneous Computingcited by 56
  14. 2020word2vec, node2vec, graph2vec, X2vec: Towards a Theory of Vector Embeddings of Structured Datacited by 38
  15. 2020funcX: A Federated Function Serving Fabric for Sciencecited by 32
  16. 2022MOFormer: Self-Supervised Transformer model for Metal-Organic Framework Property Predictioncited by 28
  17. 2023Attention Lens: A Tool for Mechanistically Interpreting the Attention Head Information Retrieval Mechanismcited by 22
  18. 2019On Neural Architecture Search for Resource-Constrained Hardware Platformscited by 18
  19. 2018DLHub: Model and Data Serving for Sciencecited by 17
  20. 2018Transfer learning to model inertial confinement fusion experimentscited by 14
  21. 2016ModelHub: Towards Unified Data and Lifecycle Management for Deep Learningcited by 13
  22. 2021QoS-Aware Placement of Deep Learning Services on the Edge with Multiple Service Implementationscited by 7

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 2.

5 papers.

  1. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 85
  2. 2024On the Foundations of Earth and Climate Foundation Modelscited by 33
  3. 2024MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AIcited by 16
  4. 2025Large Physics Models: Towards a collaborative approach with Large Language Models and Foundation Modelscited by 14
  5. 2025FrugalPrompt: Reducing Contextual Overhead in Large Language Models via Token Attributioncited by 1