# Towards Agentic Recommender Systems in the Era of Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2503.16734 · HTML: https://mlsanity.com/paper/2503.16734/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 42 of its 81 references that are in the map.
- First posted on arXiv 20 Mar 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Document understanding and multimodal RAG](https://mlsanity.com/mllm/document-understanding-and-multimodal-rag/)
- LLM agents and tool use / [Agentic recommender systems](https://mlsanity.com/agents/agentic-recommender-systems/)

## Builds on (most cited first, 30 of 42)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework](https://mlsanity.com/paper/2308.00352/) (2023)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024)
- [InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents](https://mlsanity.com/paper/2403.02691/) (2024)
- [Understanding the planning of LLM agents: A survey](https://mlsanity.com/paper/2402.02716/) (2024)
- [HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding](https://mlsanity.com/paper/2403.00425/) (2024)
- [Agent AI: Surveying the Horizons of Multimodal Interaction](https://mlsanity.com/paper/2401.03568/) (2024)
- [AutoDefense: Multi-Agent LLM Defense against Jailbreak Attacks](https://mlsanity.com/paper/2403.04783/) (2024)
- [Recommender AI Agent: Integrating Large Language Models for Interactive Recommendations](https://mlsanity.com/paper/2308.16505/) (2023)
- [RecMind: Large Language Model Powered Agent For Recommendation](https://mlsanity.com/paper/2308.14296/) (2023)
- [HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data](https://mlsanity.com/paper/2311.13614/) (2023)
- [AgentCF: Collaborative Learning with Autonomous Language Agents for Recommender Systems](https://mlsanity.com/paper/2310.09233/) (2023)
- [Retroformer: Retrospective Large Language Agents with Policy Gradient Optimization](https://mlsanity.com/paper/2308.02151/) (2023)
- [Reinforcement Knowledge Graph Reasoning for Explainable Recommendation](https://mlsanity.com/paper/1906.05237/) (2019)
- [RecSim: A Configurable Simulation Platform for Recommender Systems](https://mlsanity.com/paper/1909.04847/) (2019)
- [Personalized Prompt Learning for Explainable Recommendation](https://mlsanity.com/paper/2202.07371/) (2022)
- [User-LLM: Efficient LLM Contextualization with User Embeddings](https://mlsanity.com/paper/2402.13598/) (2024)
- [Large Language Models Empowered Personalized Web Agents](https://mlsanity.com/paper/2410.17236/) (2024)
- [Leveraging Large Language Models in Conversational Recommender Systems](https://mlsanity.com/paper/2305.07961/) (2023)
- [AgentLite: A Lightweight Library for Building and Advancing Task-Oriented LLM Agent System](https://mlsanity.com/paper/2402.15538/) (2024)
- [Let Me Do It For You: Towards LLM Empowered Recommendation via Tool Learning](https://mlsanity.com/paper/2405.15114/) (2024)
- [CoRAL: Collaborative Retrieval-Augmented Large Language Models Improve Long-tail Recommendation](https://mlsanity.com/paper/2403.06447/) (2024)
- [A Multi-Agent Conversational Recommender System](https://mlsanity.com/paper/2402.01135/) (2024)
- [LLM4Rerank: LLM-based Auto-Reranking Framework for Recommendations](https://mlsanity.com/paper/2406.12433/) (2024)
- [USimAgent: Large Language Models for Simulating Search Users](https://mlsanity.com/paper/2403.09142/) (2024)
- [AI PERSONA: Towards Life-long Personalization of LLMs](https://mlsanity.com/paper/2412.13103/) (2024)

## Built on by (most cited first, 26 of 26)

- [A Survey of Context Engineering for Large Language Models](https://mlsanity.com/paper/2507.13334/) (2025)
- [Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI](https://mlsanity.com/paper/2505.19443/) (2025)
- [AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems](https://mlsanity.com/paper/2505.19623/) (2025)
- [A Survey on Agentic Multimodal Large Language Models](https://mlsanity.com/paper/2510.10991/) (2025)
- [RecoWorld: Building Simulated Environments for Agentic Recommender Systems](https://mlsanity.com/paper/2509.10397/) (2025)
- [A Survey of Foundation Model-Powered Recommender Systems: From Feature-Based, Generative to Agentic Paradigms](https://mlsanity.com/paper/2504.16420/) (2025)
- [Skill-R1: Agent Skill Evolution via Reinforcement Learning](https://mlsanity.com/paper/2605.09359/) (2026)
- [Harness Handbook: Making Evolving Agent Harnesses Readable,Navigable, and Editable](https://mlsanity.com/paper/2607.13285/) (2026)
- [DrunkAgent: Stealthy Memory Corruption in LLM-Powered Recommender Agents](https://mlsanity.com/paper/2503.23804/) (2025)
- [Pluralistic Off-policy Evaluation and Alignment](https://mlsanity.com/paper/2509.19333/) (2025)
- [WildScore: Benchmarking MLLMs in-the-Wild Symbolic Music Reasoning](https://mlsanity.com/paper/2509.04744/) (2025)
- [In-context Ranking Preference Optimization](https://mlsanity.com/paper/2504.15477/) (2025)
- [AMPS: Adaptive Modality Preference Steering via Functional Entropy](https://mlsanity.com/paper/2602.12533/) (2026)
- [AgentWebBench: Benchmarking Multi-Agent Coordination in Agentic Web](https://mlsanity.com/paper/2604.10938/) (2026)
- [Autonomous Information Seeking: A Roadmap for Agentic Recommender Systems](https://mlsanity.com/paper/2607.04433/) (2026)
- [Factorized Latent Reasoning for LLM-based Recommendation](https://mlsanity.com/paper/2604.26760/) (2026)
- [Verifiable Reasoning for LLM-based Generative Recommendation](https://mlsanity.com/paper/2603.07725/) (2026)
- [OLIVIA: Online Learning via Inference-time Action Adaptation for Decision Making in LLM ReAct Agents](https://mlsanity.com/paper/2605.11169/) (2026)
- [Skill-CMIB: Multimodal Agent Skill for Consistent Action via Conditional Multimodal Information Bottleneck](https://mlsanity.com/paper/2605.08526/) (2026)
- [F-GRPO: Factorized Group-Relative Policy Optimization for Unified Candidate Generation and Ranking](https://mlsanity.com/paper/2605.12995/) (2026)
- [KREL: Automatic Medical Coding via Knowledge-Guided Reasoning over Clinical Evidence with LLMs](https://mlsanity.com/paper/2608.20887/) (2026)
- [To See or To Read: User Behavior Reasoning in Multimodal LLMs](https://mlsanity.com/paper/2511.03845/) (2025)
- [AgenticRec: A Recommendation-Oriented Agentic Framework with Progressive Tool-Integrated Reasoning Optimization](https://mlsanity.com/paper/2603.21613/) (2026)
- [Fair Agents: Balancing Multistakeholder Alignment in Multi-Agent Personalization Systems](https://mlsanity.com/paper/2605.02379/) (2026)
- [RecoAtlas: From Semantic Plausibility to Set-Level Utility in LLM Recommendation Agents](https://mlsanity.com/paper/2605.18805/) (2026)
- [Personal-Agent Mediated Recommendation with Cross-Platform User History](https://mlsanity.com/paper/2610.07588/) (2026)

## Abstract (arXiv)

Recent breakthroughs in Large Language Models (LLMs) have led to the emergence of agentic AI systems that extend beyond the capabilities of standalone models. By empowering LLMs to perceive external environments, integrate multimodal information, and interact with various tools, these agentic systems exhibit greater autonomy and adaptability across complex tasks. This evolution brings new opportunities to recommender systems (RS): LLM-based Agentic RS (LLM-ARS) can offer more interactive, context-aware, and proactive recommendations, potentially reshaping the user experience and broadening the application scope of RS. Despite promising early results, fundamental challenges remain, including how to effectively incorporate external knowledge, balance autonomy with controllability, and evaluate performance in dynamic, multimodal settings. In this perspective paper, we first present a systematic analysis of LLM-ARS: (1) clarifying core concepts and architectures; (2) highlighting how agentic capabilities -- such as planning, memory, and multimodal reasoning -- can enhance recommendation quality; and (3) outlining key research questions in areas such as safety, efficiency, and lifelong personalization. We also discuss open problems and future directions, arguing that LLM-ARS will drive the next wave of RS innovation. Ultimately, we foresee a paradigm shift toward intelligent, autonomous, and collaborative recommendation experiences that more closely align with users' evolving needs and complex decision-making processes.