# Design and evaluation of AI copilots -- case studies of retail copilot templates

arXiv: https://arxiv.org/abs/2407.09512 · HTML: https://mlsanity.com/paper/2407.09512/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 26 references that are in the map.
- First posted on arXiv 17 Jun 2024.

## Builds on (most cited first, 10 of 10)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs](https://mlsanity.com/paper/2312.05934/) (2023)
- [Chain of Natural Language Inference for Reducing Large Language Model Ungrounded Hallucinations](https://mlsanity.com/paper/2310.03951/) (2023)
- [Building Your Own Product Copilot: Challenges, Opportunities, and Needs](https://mlsanity.com/paper/2312.14231/) (2023)
- [The Rise of the AI Co-Pilot: Lessons for Design from Aviation and Beyond](https://mlsanity.com/paper/2311.14713/) (2023)
- [A State-of-the-practice Release-readiness Checklist for Generative AI-based Software Products](https://mlsanity.com/paper/2403.18958/) (2024)

## Built on by (most cited first, 3 of 3)

- [Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts](https://mlsanity.com/paper/2409.12447/) (2024)
- [Modeling and Optimizing User Preferences in AI Copilots: A Comprehensive Survey and Taxonomy](https://mlsanity.com/paper/2505.21907/) (2025)
- [PeerCoPilot: A Language Model-Powered Assistant for Behavioral Health Organizations](https://mlsanity.com/paper/2511.21721/) (2025)

## Abstract (arXiv)

Building a successful AI copilot requires a systematic approach. This paper is divided into two sections, covering the design and evaluation of a copilot respectively. A case study of developing copilot templates for the retail domain by Microsoft is used to illustrate the role and importance of each aspect. The first section explores the key technical components of a copilot's architecture, including the LLM, plugins for knowledge retrieval and actions, orchestration, system prompts, and responsible AI guardrails. The second section discusses testing and evaluation as a principled way to promote desired outcomes and manage unintended consequences when using AI in a business context. We discuss how to measure and improve its quality and safety, through the lens of an end-to-end human-AI decision loop framework. By providing insights into the anatomy of a copilot and the critical aspects of testing and evaluation, this paper provides concrete evidence of how good design and evaluation practices are essential for building effective, human-centered AI assistants.