mlsanityData to 14 Oct 2026

Software Performance Engineering for Foundation Model-Powered Software

arXiv 2411.09580 ↗First posted 14 Nov 2024cs.SE cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

The rise of Foundation Models (FMs) like Large Language Models (LLMs) is revolutionizing software development. Despite the impressive prototypes, transforming FMware into production-ready products demands complex engineering across various domains. A critical but overlooked aspect is performance engineering, which aims at ensuring FMware meets performance goals such as throughput and latency to avoid user dissatisfaction and financial loss. Often, performance considerations are an afterthought, leading to costly optimization efforts post-deployment. FMware's high computational resource demands highlight the need for efficient hardware use. Continuous performance engineering is essential to prevent degradation. This paper highlights the significance of Software Performance Engineering (SPE) in FMware, identifying four key challenges: cognitive architecture design (i.e., the structural design that defines how AI components interact, reason, and interface with classical software components), communication protocols, tuning and optimization, and deployment. These challenges are based on literature surveys and experiences from developing an in-house FMware system. We discuss problems, current practices, and innovative paths for the software engineering community.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 48 of its 96 references that are in the map. First posted on arXiv 14 Nov 2024.

Builds on

48 of 96 references.

References · 48

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  5. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  6. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  7. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 769
  8. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 742
  9. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  10. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 560
  11. 2023Cognitive Architectures for Language Agentscited by 429
  12. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
36 more
  1. 2024RepairAgent: An Autonomous, LLM-Based Agent for Program Repaircited by 232
  2. 2023Language Model Tokenizers Introduce Unfairness Between Languagescited by 214
  3. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  4. 2023SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefillscited by 193
  5. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 174
  6. 2023Speculative Decoding with Big Little Decodercited by 164
  7. 2024LLM Inference Unveiled: Survey and Roofline Model Insightscited by 152
  8. 2023Fast Distributed Inference Serving for Large Language Modelscited by 145
  9. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  10. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 120
  11. 2024Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generationcited by 101
  12. 2024Parrot: Efficient Serving of LLM-based Applications with Semantic Variablecited by 99
  13. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  14. 2023BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agentscited by 84
  15. 2023AutoML-GPT: Automatic Machine Learning with GPTcited by 81
  16. 2023ISR-LLM: Iterative Self-Refined Large Language Model for Long-Horizon Sequential Task Planningcited by 77
  17. 2024Beyond Efficiency: A Systematic Survey of Resource-Efficient Large Language Modelscited by 67
  18. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  19. 2024Towards Greener LLMs: Bringing Energy-Efficiency to the Forefront of LLM Inferencecited by 61
  20. 2024Understanding LLMs: A Comprehensive Overview from Training to Inferencecited by 59
  21. 2024Understanding the Performance and Estimating the Cost of LLM Fine-Tuningcited by 48
  22. 2024"We Need Structured Output": Towards User-centered Constraints on Large Language Model Outputcited by 46
  23. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 45
  24. 2024Towards Optimizing the Costs of LLM Usagecited by 33
  25. 2024M'elange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneitycited by 31
  26. 2024LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMscited by 25
  27. 2022Towards Training Reproducible Deep Learning Modelscited by 23
  28. 2024GPT Semantic Cache: Reducing LLM Costs and Latency via Semantic Embedding Cachingcited by 22
  29. 2024Enabling Efficient Batch Serving for LMaaS via Generation Length Predictioncited by 20
  30. 2024AutoMMLab: Automatically Generating Deployable Models from Language Instructions for Computer Vision Taskscited by 14
  31. 2024Beyond Natural Language: LLMs Leveraging Alternative Formats for Enhanced Reasoning and Communicationcited by 14
  32. 2024LLM-Pilot: Characterize and Optimize Performance of your LLM Inference Servicescited by 14
  33. 2024CEBench: A Benchmarking Toolkit for the Cost-Effectiveness of LLM Pipelinescited by 11
  34. 2024SelectLLM: Query-Aware Efficient Selection Algorithm for Large Language Modelscited by 8
  35. 2024Cost-Performance Optimization for Processing Low-Resource Language Tasks Using Commercial LLMscited by 7
  36. 2024Revisiting Service Level Objectives and System Level Metrics in Large Language Model Servingcited by 5

Built on by

Not cited yet.