mlsanityData to 7 Oct 2026

AURA: Agentic Diagnosis and Refinement for Production Recommender Systems at Scale

arXiv 2609.16625 ↗First posted 15 Sep 2026cs.IR cs.AI cs.LGcited by 0

On the map: LLM agents and tool use → Agentic recommender systems

Abstract (arXiv)

How and why does a recommender system fail the users it serves? Oftentimes, practitioners are left to improve their algorithms based on a combination of feedback from stakeholder teams, domain expertise, and insights from data analyses. Yet the nuances of how and where recommendations perform well or poorly for end users are difficult to discern from aggregate quantitative metrics. Whereas these metrics provide a high-level and incomplete picture, further granularity into the quality of recommendations and their patterns requires reasoning with domain understanding and objectivity, at scale. We contemplate this complex conundrum and describe a method and implementation that uses the latest AI agentic advances to provide actionable diagnoses and improvements for production recommender systems. We present AURA (Agentic Understanding and Refinement of recommender Algorithms), an end-to-end agentic system that performs qualitative evaluation at scale and can then generate improvements to our algorithms at the code level. Specialized agents read production engagement logs, from thousands of sessions to millions, and surface patterns and examples of how the recommender fails real users. The next step uses those diagnoses and context about the recommender's own code, data, and training pipeline to propose and implement refinements grounded in that codebase. We report the system design, initial tests on production data from two large consumer platforms at a major media-streaming company, safeguards, operational learnings, and early results toward a self-improving recommender system. Finally, the diagnostic gap AURA closes is not specific to streaming. The architecture is built to transfer: every domain-specific element enters through the configuration layer that already ported it between our two platforms. We map it concretely to e-commerce and online-retail recommendation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Agentic recommender systems, one of 16 directions in LLM agents and tool use. Builds on 25 of its 34 references that are in the map. First posted on arXiv 15 Sep 2026.

Builds on

25 of 34 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Self-Evolving Recommendation System: End-To-End Autonomous Model Optimization With LLM Agents10 Feb
  2. 2025No-Human in the Loop: Agentic Evaluation at Scale for Recommendation4 Nov
  3. 2023On Generative Agents in Recommendation16 Oct

Other references · 22

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  3. 2018Model Cards for Model Reportingcited by 1,068
  4. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  5. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  6. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  7. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
  8. 2023MLAgentBench: Evaluating Language Agents on Machine Learning Experimentationcited by 339
  9. 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 307
  10. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  11. 2023User Behavior Simulation with Large Language Model based Agentscited by 136
  12. 2017Beyond Parity: Fairness Objectives for Collaborative Filteringcited by 93
10 more
  1. 2023Human Feedback is not Gold Standardcited by 82
  2. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  3. 2018Automated Data Slicing for Model Validation:A Big data - AI Integration Approachcited by 70
  4. 2023Style Over Substance: Evaluation Biases for Large Language Modelscited by 68
  5. 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 50
  6. 2023Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Modelscited by 48
  7. 2024RecAI: Leveraging Large Language Models for Next-Generation Recommender Systemscited by 20
  8. 2021The Simpson's Paradox in the Offline Evaluation of Recommendation Systemscited by 16
  9. 2021Beyond NDCG: behavioral testing of recommender systems with RecListcited by 8
  10. 2026Self-EvolveRec: Self-Evolving Recommender Systems with LLM-based Directional Feedbackcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CORAL: An LLM-Native Harness for Production Recommender Systems
  2. 2026Rethinking Recommendation Paradigms: From Pipelines to Agentic Recommender Systems
  3. 2026Who Are We Recommending To? Recommender Systems in the Agentic Web
  4. 2026Autonomous Information Seeking: A Roadmap for Agentic Recommender Systems
  5. 2025AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems
  6. 2025The Future is Agentic: Definitions, Perspectives, and Open Challenges of Multi-Agent Recommender Systems
  7. 2026EvoRec: Self Evolving Agentic Recommender Systems
  8. 2026RecEvolve: A Knowledge-Driven Autonomous Agent System for Recommender Systems