mlsanityData to 14 Oct 2026

From Cool Demos to Production-Ready FMware: Core Challenges and a Technology Roadmap

arXiv 2410.20791 ↗First posted 28 Oct 2024cs.SE cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

The rapid expansion of foundation models (FMs), such as large language models (LLMs), has given rise to FMware, software systems that integrate FM(s) as core components. While building demonstration-level FMware is relatively straightforward, transitioning to production-ready systems presents numerous challenges, including reliability, high implementation costs, scalability, and compliance with privacy regulations. Our paper conducts a semi-structured thematic synthesis to identify key challenges in productionizing FMware across diverse data sources, including our industry experience developing FMArts, a FMware lifecycle engineering platform, and its integration into Huawei Cloud; grey literature; academic publications; hands-on involvement in the Open Platform for Enterprise AI (OPEA); organizing the AIware conference and bootcamp; and co-leading the ISO SPDX SBOM working group on AI and datasets. We identify critical issues in FM(s) selection, data and model alignment, prompt engineering, agent orchestration, system testing, and deployment, alongside cross-cutting concerns such as memory management, observability, and feedback integration. We discuss necessary technologies and strategies to address these challenges and offer guidance to enable the transition from demonstration systems to scalable, production-ready FMware solutions. Our findings underscore the importance of continued research and multi-industry collaboration to advance the development of production-ready FMware.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 65 of its 114 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

65 of 114 references.

References · 65

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  3. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  4. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  5. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  6. 2018Datasheets for Datasetscited by 1,386
  7. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  8. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 769
  9. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  10. 2024Understanding the planning of LLM agents: A surveycited by 421
  11. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
  12. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 365
38 more
  1. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 353
  2. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 341
  3. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  4. 2024The Power of Noise: Redefining Retrieval for RAG Systemscited by 281
  5. 2023Large Language Model Alignment: A Surveycited by 243
  6. 2024A Survey on Data Selection for Language Modelscited by 236
  7. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 205
  8. 2023Efficient Large Language Models: A Surveycited by 186
  9. 2023Benchmarking Cognitive Biases in Large Language Models as Evaluatorscited by 150
  10. 2022Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidencecited by 144
  11. 2022PromptChainer: Chaining Large Language Model Prompts through Visual Programmingcited by 134
  12. 2023Accelerating Transformer Inference for Translation via Parallel Decodingcited by 124
  13. 2024Best Practices and Lessons Learned on Synthetic Datacited by 106
  14. 2023Regulating ChatGPT and other Large Generative AI Modelscited by 104
  15. 2022Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Modelscited by 97
  16. 2024CodeR: Issue Resolving with Multi-Agent and Task Graphscited by 91
  17. 2024Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systemscited by 87
  18. 2024Blended RAG: Improving RAG (Retriever-Augmented Generation) Accuracy with Semantic Search and Hybrid Query-Based Retrieverscited by 84
  19. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 73
  20. 2024The Ultimate Guide to Fine-Tuning LLMs from Basics to Breakthroughs: An Exhaustive Review of Technologies, Research, Best Practices, Applied Research Challenges and Opportunitiescited by 64
  21. 2023Learning to Retrieve In-Context Examples for Large Language Modelscited by 60
  22. 2023Maybe Only 0.5% Data is Needed: A Preliminary Exploration of Low Training Data Instruction Tuningcited by 58
  23. 2023A Bibliometric Review of Large Language Models Research from 2017 to 2023cited by 57
  24. 2024Evaluating LLMs at Detecting Errors in LLM Responsescited by 50
  25. 2024Understanding the Performance and Estimating the Cost of LLM Fine-Tuningcited by 48
  26. 2024Efficient Prompting Methods for Large Language Models: A Surveycited by 47
  27. 2024Current state of LLM Risks and AI Guardrailscited by 46
  28. 2024LLM Inference Serving: Survey of Recent Advances and Opportunitiescited by 46
  29. 2024How Easily do Irrelevant Inputs Skew the Responses of Large Language Models?cited by 44
  30. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  31. 2023A Survey on ChatGPT: AI-Generated Contents, Challenges, and Solutionscited by 35
  32. 2022Validating Large Language Models with ReLMcited by 33
  33. 2024Towards AI-Native Software Engineering (SE 3.0): A Vision and a Challenge Roadmapcited by 30
  34. 2023Better patching using LLM prompting, via Self-Consistencycited by 29
  35. 2024Arena Learning: Build Data Flywheel for LLMs Post-training via Simulated Chatbot Arenacited by 26
  36. 2023(Why) Is My Prompt Getting Worse? Rethinking Regression Testing for Evolving LLM APIscited by 23
  37. 2023KnowledgeNavigator: Leveraging Large Language Models for Enhanced Reasoning over Knowledge Graphcited by 22
  38. 2024Adapting LLMs for Efficient Context Processing through Soft Prompt Compressioncited by 22

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025An Empirical Study of Testing Practices in Open Source AI Agent Frameworks and Agentic Applicationscited by 15
  2. 2026Agent-Integrated Software: Interaction Contracts and Continuous Assurancecited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Rethinking Software Engineering in the Foundation Model Era: A Curated Catalogue of Challenges in the Development of Trustworthy FMware
  2. 2026Evaluating Local Language Model Agents for Reproducible Data Engineering: An Empirical Software Engineering Study of Mobility Workflows
  3. 2026A Large-Scale Study on the Development and Issues of Multi-Agent AI Systems
  4. 2026AI Observability for Large Language Model Systems: A Multi-Layer Analysis of Monitoring Approaches from Confidence Calibration to Infrastructure Tracing ↗
  5. 2026Model Context Protocol (MCP) Tool Descriptions Are Smelly! Towards Improving AI Agent Efficiency with Augmented MCP Tool Descriptions
  6. 2025The FM Agent
  7. 2026Advanced AI Service Provisioning in O-RAN through LLM Engine Integration
  8. 2024The Fusion of Large Language Models and Formal Methods for Trustworthy AI Agents: A Roadmap