mlsanityData to 14 Oct 2026

DeepSpeed4Science Initiative: Enabling Large-Scale Scientific Discovery through Sophisticated AI System Technologies

arXiv 2310.04610 ↗First posted 6 Oct 2023cs.AI cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

In the upcoming decade, deep learning may revolutionize the natural sciences, enhancing our capacity to model and predict natural occurrences. This could herald a new era of scientific exploration, bringing significant advancements across sectors from drug development to renewable energy. To answer this call, we present DeepSpeed4Science initiative (deepspeed4science.ai) which aims to build unique capabilities through AI system technology innovations to help domain experts to unlock today's biggest science mysteries. By leveraging DeepSpeed's current technology pillars (training, inference and compression) as base technology enablers, DeepSpeed4Science will create a new set of AI system technologies tailored for accelerating scientific discoveries by addressing their unique complexity beyond the common technical approaches used for accelerating generic large language models (LLMs). In this paper, we showcase the early progress we made with DeepSpeed4Science in addressing two of the critical system challenges in structural biology research.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 8 of its 14 references that are in the map. First posted on arXiv 6 Oct 2023.

Builds on

8 of 14 references.

References · 8

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  5. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  6. 2023ClimaX: A foundation model for weather and climatecited by 337
  7. 2021Do Transformers Really Perform Bad for Graph Representation?cited by 320
  8. 2021MS-nowcasting: Operational Precipitation Nowcasting with Convolutional LSTMs at Microsoft Weathercited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 7; 2026: 1.

15 papers.

Sort:

By area:

  1. 2024Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encodingcited by 72
  2. 2024EVF-SAM: Early Vision-Language Fusion for Text-Prompted Segment Anything Modelcited by 65
  3. 2025Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUscited by 22
  4. 2024DataStates-LLM: Lazy Asynchronous Checkpointing for Large Language Modelscited by 16
  5. 2024SmileyLlama: Modifying Large Language Models for Directed Chemical Space Explorationcited by 14
  6. 2025Advancing the Scientific Method with Large Language Models: From Hypothesis to Discoverycited by 7
  7. 2024TSAK: Two-Stage Semantic-Aware Knowledge Distillation for Efficient Wearable Modality and Model Optimization in Manufacturing Linescited by 5
  8. 20247B Fully Open Source Moxin-LLM/VLM -- From Pretraining to GRPO-based Reinforcement Learning Enhancementcited by 5
  9. 2025MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Modelscited by 4
  10. 2024Activating Distributed Visual Region within LLMs for Efficient and Effective Vision-Language Training and Inferencecited by 3
  11. 2025A Survey on Memory-Efficient Transformer-Based Model Training in AI for Sciencecited by 3
  12. 2025FlashRecovery: Fast and Low-Cost Recovery from Failures for Large-Scale Training of LLMscited by 2
3 more
  1. 2025MegaFold: Efficient Training of Next-Generation 3D Attention Protein Models on Cross-Platform GPUscited by 1
  2. 2025MLP-Offload: Multi-Level, Multi-Path Offloading for LLM Pre-training to Break the GPU Memory Wallcited by 0
  3. 2026TurboPairFormer: Fast and Stable Protein Folding Model Training with an Optimized Triangle Attention Kernelcited by 0