# PromptPack: Scaling LLM Annotation Agents for Online Recommendation

arXiv: https://arxiv.org/abs/2607.20528 · HTML: https://mlsanity.com/paper/2607.20528/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agentic recommender systems, one of 16 directions in LLM agents and tool use.
- Builds on 7 of its 15 references that are in the map.
- First posted on arXiv 10 Jul 2026.

## Placed in

- LLM agents and tool use / [Agentic recommender systems](https://mlsanity.com/agents/agentic-recommender-systems/)

## Builds on (most cited first, 7 of 7)

- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [Batch Prompting: Efficient Inference with Large Language Model APIs](https://mlsanity.com/paper/2301.08721/) (2023)
- [BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batching](https://mlsanity.com/paper/2412.03594/) (2024)
- [CliqueParcel: An Approach For Batching LLM Prompts That Jointly Optimizes Efficiency And Faithfulness](https://mlsanity.com/paper/2402.14833/) (2024)
- [Real-time Ad retrieval via LLM-generative Commercial Intention for Sponsored Search Advertising](https://mlsanity.com/paper/2504.01304/) (2025)
- [XML Prompting as Grammar-Constrained Interaction: Fixed-Point Semantics, Convergence Guarantees, and Human-AI Protocols](https://mlsanity.com/paper/2509.08182/) (2025)
- [Agent0: Leveraging LLM Agents to Discover Multi-value Features from Text for Enhanced Recommendations](https://mlsanity.com/paper/2507.18993/) (2025)

## Abstract (arXiv)

Online recommendation platforms increasingly use Large Language Models (LLMs) to extract structured features from ad creatives. While deploying a single-call LLM annotation agent yields significant Click-Through Rate (CTR) improvements in our live production environment, per-creative prompting is prohibitively expensive to scale. The redundant system instructions sent in every request account for 94% of billed input tokens. To break this cost bottleneck, we introduce PromptPack, a scalable, high-throughput LLM annotation agent. PromptPack achieves this scale via in-context batching, combining a shared system prompt, a strict XML structural envelope, and an output correction layer to ensure deterministic, pipeline-ready feature extraction across multiple creatives simultaneously. We evaluate PromptPack via an offline retrieval benchmark using a downstream logistic-regression ranker. To deeply profile the agent's behavior, we measure AUC and introduce Volume-Weighted Absolute Lift (VWAL), a novel metric capturing the signal quality of the generated features. Compared to our live, unbatched production baseline, PromptPack at batch size 20 cuts our LLM costs by 89% and accelerates throughput by 2.5x while fully preserving AUC.