mlsanityData to 14 Oct 2026

A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models

arXiv 2410.07265 ↗First posted 8 Oct 2024cs.AR cs.AI cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

The rapid development of large language models (LLMs) has significantly transformed the field of artificial intelligence, demonstrating remarkable capabilities in natural language processing and moving towards multi-modal functionality. These models are increasingly integrated into diverse applications, impacting both research and industry. However, their development and deployment present substantial challenges, including the need for extensive computational resources, high energy consumption, and complex software optimizations. Unlike traditional deep learning systems, LLMs require unique optimization strategies for training and inference, focusing on system-level efficiency. This paper surveys hardware and software co-design approaches specifically tailored to address the unique characteristics and constraints of large language models. This survey analyzes the challenges and impacts of LLMs on hardware and algorithm research, exploring algorithm optimization, hardware design, and system-level innovations. It aims to provide a comprehensive understanding of the trade-offs and considerations in LLM-centric computing systems, guiding future advancements in AI. Finally, we summarize the existing efforts in this space and outline future directions toward realizing production-grade co-design methodologies for the next generation of large language models and AI systems.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 157 of its 232 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

157 of 232 references.

References · 157

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2023GPT-4 Technical Reportcited by 18,396
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  9. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  10. 2023Visual Instruction Tuningcited by 7,869
  11. 2020Scaling Laws for Neural Language Modelscited by 6,586
  12. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
38 more
  1. 2016Layer Normalizationcited by 5,669
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  3. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  4. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  7. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  8. 2020Longformer: The Long-Document Transformercited by 3,413
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  10. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  11. 2023A Survey of Large Language Modelscited by 2,576
  12. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  13. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  14. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  15. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  16. 2024Mixtral of Expertscited by 1,823
  17. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  18. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  19. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  20. 2020Rethinking Attention with Performerscited by 1,601
  21. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  22. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  23. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  24. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  25. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  26. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  27. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  28. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  29. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  30. 2017Mixed Precision Trainingcited by 1,156
  31. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  32. 2021Perceiver: General Perception with Iterative Attentioncited by 1,104
  33. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  34. 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 861
  35. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  36. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  37. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  38. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717

50 most cited of 157 All 157 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 15; 2026: 8.

23 papers.

Sort:

By area:

  1. 2025AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challengescited by 169
  2. 2025M-ANT: Efficient Low-bit Group Quantization for LLMs via Mathematically Adaptive Numerical Typecited by 16
  3. 2025A Survey on Inference Engines for Large Language Models: Perspectives on Optimization and Efficiencycited by 12
  4. 2025Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directionscited by 10
  5. 2025AutoEDA: Enabling EDA Flow Automation through Microservice-Based LLM Agentscited by 6
  6. 2025lm-Meter: Unveiling Runtime Inference Latency for On-Device Language Modelscited by 5
  7. 2026Hardware Co-Design Scaling Laws via Roofline Modelling for On-Device LLMscited by 5
  8. 2026A Survey on the Green Development of Large Models: From Resource-Efficient Architectures to Hardware-Software Co-Designcited by 5
  9. 2025A Survey on Memory-Efficient Transformer-Based Model Training in AI for Sciencecited by 3
  10. 2025Can AI automatically analyze public opinion? A LLM agents-based agentic pipeline for timely public opinion analysiscited by 2
  11. 2025OASIS: Outlier-Aware LUT-Based GEMM with Dual-Side Quantization for LLM Inference Accelerationcited by 2
  12. 2025Platinum: Path-Adaptable LUT-Based Accelerator Tailored for Low-Bit Weight Matrix Multiplicationcited by 2
11 more
  1. 2026Agentic Agile-V: From Vibe Coding to Verified Engineering in Software and Hardware Developmentcited by 2
  2. 2025Beyond Moore's Law: Harnessing the Redshift of Generative AI with Effective Hardware-Software Co-Designcited by 1
  3. 2025AI Compute Architecture and Evolution Trendscited by 1
  4. 2025Bhasha-Rupantarika: Algorithm-Hardware Co-design approach for Multilingual Neural Machine Translationcited by 0
  5. 2025A Vision Toward Energy-Efficient Domain-Specific Artificial Intelligence Models and Agentscited by 0
  6. 2025CAMformer: Associative Memory is All You Needcited by 0
  7. 2026Joint Hardware-Workload Co-Optimization for In-Memory Computing Acceleratorscited by 0
  8. 2026Tempus: A Temporally Scalable Resource-Invariant GEMM Streaming Framework for Versal AI Edgecited by 0
  9. 2026NASiC: 3D NAND-based CAM-Selected Multibit CIM Architecture for Efficient On-Device Mixture-of-Experts LLM Inferencecited by 0
  10. 2026Gradient Transformer: Learning to Generate Updates for LLMscited by 0
  11. 2026Deep Learning for Anomaly Detection in Railway Systems: A Structured Surveycited by 0