mlsanityData to 14 Oct 2026

A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Design

arXiv 2607.09084 ↗First posted 10 Jul 2026cs.LG cs.CYcited by 5

Not on the map. Why?

Abstract (arXiv)

The rapid expansion of large-scale AI models has led to significant performance breakthroughs across diverse domains, yet it has also raised critical concerns regarding computational costs, energy consumption, and environmental sustainability. This survey provides a comprehensive overview of the green development of large models, emphasizing resource-efficient architectures and full-stack hardware-software co-design. We systematically review recent advances in efficient model construction, including attention operator optimization, linear-complexity architectures, and model sparsification and merging, as well as training and deployment strategies such as data-efficient learning, parameter-efficient fine-tuning, and computational compression. Beyond algorithmic improvements, we explore energy-efficient AI hardware, including mainstream AI chips, memory optimization, cross-platform deployment, and sustainable infrastructure. Furthermore, we examine how large models are being applied to sustainability-critical domains such as DeepSeek, remote sensing interpretation, national-scale infrastructure, and global initiatives. Finally, we discuss key challenges and future directions, highlighting the need for continual learning paradigms, memory-centric hardware, and standardized evaluation protocols. This survey aims to offer a holistic roadmap toward sustainable, scalable, and socially responsible development of large models. Paper homepage: https://cje.ejournal.org.cn/article/doi/10.23919/cje.2025.00.438

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 142 of its 195 references that are in the map. First posted on arXiv 10 Jul 2026.

Builds on

142 of 195 references.

References · 142

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  9. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  10. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  11. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  12. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
38 more
  1. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  5. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 2,022
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  8. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  10. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  11. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  12. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  13. 2017Mixed Precision Trainingcited by 1,156
  14. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  15. 2022Editing Models with Task Arithmeticcited by 1,120
  16. 2024VMamba: Visual State Space Modelcited by 969
  17. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  18. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 867
  19. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  20. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  21. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  22. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  23. 2021GPT Understands, Toocited by 716
  24. 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 678
  25. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 637
  26. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  27. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  28. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  29. 2021Deep Learning on a Data Diet: Finding Important Examples Early in Trainingcited by 533
  30. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  31. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 502
  32. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  33. 2022Dataset Distillation by Matching Training Trajectoriescited by 442
  34. 2024SliceGPT: Compress Large Language Models by Deleting Rows and Columnscited by 385
  35. 2021Dataset Condensation with Distribution Matchingcited by 365
  36. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 328
  37. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 324
  38. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300

50 most cited of 142 All 142 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026UniMPA: A Unified Memory-Prediction-Action Model via Action-Grounded Transition Modelingcited by 1
  2. 2026Learning to Use Imagination: Progress-Conditioned Future Utilization for World Action Modelscited by 0
  3. 2026ATI-VLA: Action-Centric Predictive Vision-Language-Action Models via Actionable Alignment Then Adaptive Injectioncited by 0
  4. 2026Why DiT Models Underperform as Representation Learners without Long Skip Connectionscited by 0
  5. 2026Narrative Paradoxes in LLM Inference: Shaping Reasoning Trajectories toward Misalignmentcited by 0