mlsanityData to 14 Oct 2026

PromptPack: Scaling LLM Annotation Agents for Online Recommendation

arXiv 2607.20528 ↗First posted 10 Jul 2026cs.AIcited by 0

On the map: LLM agents and tool use → Agentic recommender systems

Abstract (arXiv)

Online recommendation platforms increasingly use Large Language Models (LLMs) to extract structured features from ad creatives. While deploying a single-call LLM annotation agent yields significant Click-Through Rate (CTR) improvements in our live production environment, per-creative prompting is prohibitively expensive to scale. The redundant system instructions sent in every request account for 94% of billed input tokens. To break this cost bottleneck, we introduce PromptPack, a scalable, high-throughput LLM annotation agent. PromptPack achieves this scale via in-context batching, combining a shared system prompt, a strict XML structural envelope, and an output correction layer to ensure deterministic, pipeline-ready feature extraction across multiple creatives simultaneously. We evaluate PromptPack via an offline retrieval benchmark using a downstream logistic-regression ranker. To deeply profile the agent's behavior, we measure AUC and introduce Volume-Weighted Absolute Lift (VWAL), a novel metric capturing the signal quality of the generated features. Compared to our live, unbatched production baseline, PromptPack at batch size 20 cuts our LLM costs by 89% and accelerates throughput by 2.5x while fully preserving AUC.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agentic recommender systems, one of 16 directions in LLM agents and tool use. Builds on 7 of its 15 references that are in the map. First posted on arXiv 10 Jul 2026.

Builds on

7 of 15 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Agent0: Leveraging LLM Agents to Discover Multi-value Features from Text for Enhanced Recommendations25 Jul

Other references · 6

  1. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  2. 2023Batch Prompting: Efficient Inference with Large Language Model APIscited by 85
  3. 2024BatchLLM: Optimizing Large Batched LLM Inference with Global Prefix Sharing and Throughput-oriented Token Batchingcited by 32
  4. 2024CliqueParcel: An Approach For Batching LLM Prompts That Jointly Optimizes Efficiency And Faithfulnesscited by 12
  5. 2025Real-time Ad retrieval via LLM-generative Commercial Intention for Sponsored Search Advertisingcited by 8
  6. 2025XML Prompting as Grammar-Constrained Interaction: Fixed-Point Semantics, Convergence Guarantees, and Human-AI Protocolscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting Prompt Engineering: A Comprehensive Evaluation for LLM-based Personalized Recommendation
  2. 2025Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systems
  3. 2026Less is More: Benchmarking LLM Based Recommendation Agents
  4. 2025Language Ranker: A Lightweight Ranking framework for LLM Decoding
  5. 2026Self-Evolving Recommendation System: End-To-End Autonomous Model Optimization With LLM Agents
  6. 2026Toollery: Scaling LLM Agents to Thousands of Skills and Tools
  7. 2024Retrieve, Annotate, Evaluate, Repeat: Leveraging Multimodal LLMs for Large-Scale Product Retrieval Evaluation
  8. 2026Personalized LLM Recommendation with Semantic IDs: Autoregressive Beam Search vs. Multi-Token Prediction