# From Cool Demos to Production-Ready FMware: Core Challenges and a Technology Roadmap

arXiv: https://arxiv.org/abs/2410.20791 · HTML: https://mlsanity.com/paper/2410.20791/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 65 of its 114 references that are in the map.
- First posted on arXiv 28 Oct 2024.

## Builds on (most cited first, 30 of 65)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [Understanding the planning of LLM agents: A survey](https://mlsanity.com/paper/2402.02716/) (2024)
- [Large Language Models for Software Engineering: A Systematic Literature Review](https://mlsanity.com/paper/2308.10620/) (2023)
- [NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails](https://mlsanity.com/paper/2310.10501/) (2023)
- [Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback](https://mlsanity.com/paper/2302.12813/) (2023)
- [Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts](https://mlsanity.com/paper/2305.13300/) (2023)
- [Humans or LLMs as the Judge? A Study on Judgement Biases](https://mlsanity.com/paper/2402.10669/) (2024)
- [The Power of Noise: Redefining Retrieval for RAG Systems](https://mlsanity.com/paper/2401.14887/) (2024)
- [Large Language Model Alignment: A Survey](https://mlsanity.com/paper/2309.15025/) (2023)
- [A Survey on Data Selection for Language Models](https://mlsanity.com/paper/2402.16827/) (2024)
- [Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges](https://mlsanity.com/paper/2406.12624/) (2024)
- [Efficient Large Language Models: A Survey](https://mlsanity.com/paper/2312.03863/) (2023)
- [Benchmarking Cognitive Biases in Large Language Models as Evaluators](https://mlsanity.com/paper/2309.17012/) (2023)
- [Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence](https://mlsanity.com/paper/2210.13701/) (2022)
- [PromptChainer: Chaining Large Language Model Prompts through Visual Programming](https://mlsanity.com/paper/2203.06566/) (2022)
- [Accelerating Transformer Inference for Translation via Parallel Decoding](https://mlsanity.com/paper/2305.10427/) (2023)
- [Best Practices and Lessons Learned on Synthetic Data](https://mlsanity.com/paper/2404.07503/) (2024)
- [Regulating ChatGPT and other Large Generative AI Models](https://mlsanity.com/paper/2302.02337/) (2023)
- [Prompt Compression and Contrastive Conditioning for Controllability and Toxicity Reduction in Language Models](https://mlsanity.com/paper/2210.03162/) (2022)
- [CodeR: Issue Resolving with Multi-Agent and Task Graphs](https://mlsanity.com/paper/2406.01304/) (2024)
- [Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systems](https://mlsanity.com/paper/2407.01370/) (2024)
- [Blended RAG: Improving RAG (Retriever-Augmented Generation) Accuracy with Semantic Search and Hybrid Query-Based Retrievers](https://mlsanity.com/paper/2404.07220/) (2024)

## Built on by (most cited first, 2 of 2)

- [An Empirical Study of Testing Practices in Open Source AI Agent Frameworks and Agentic Applications](https://mlsanity.com/paper/2509.19185/) (2025)
- [Agent-Integrated Software: Interaction Contracts and Continuous Assurance](https://mlsanity.com/paper/2609.11381/) (2026)

## Abstract (arXiv)

The rapid expansion of foundation models (FMs), such as large language models (LLMs), has given rise to FMware, software systems that integrate FM(s) as core components. While building demonstration-level FMware is relatively straightforward, transitioning to production-ready systems presents numerous challenges, including reliability, high implementation costs, scalability, and compliance with privacy regulations. Our paper conducts a semi-structured thematic synthesis to identify key challenges in productionizing FMware across diverse data sources, including our industry experience developing FMArts, a FMware lifecycle engineering platform, and its integration into Huawei Cloud; grey literature; academic publications; hands-on involvement in the Open Platform for Enterprise AI (OPEA); organizing the AIware conference and bootcamp; and co-leading the ISO SPDX SBOM working group on AI and datasets. We identify critical issues in FM(s) selection, data and model alignment, prompt engineering, agent orchestration, system testing, and deployment, alongside cross-cutting concerns such as memory management, observability, and feedback integration. We discuss necessary technologies and strategies to address these challenges and offer guidance to enable the transition from demonstration systems to scalable, production-ready FMware solutions. Our findings underscore the importance of continued research and multi-industry collaboration to advance the development of production-ready FMware.