# AgentSociety Challenge: Designing LLM Agents for User Modeling and Recommendation on Web Platforms

arXiv: https://arxiv.org/abs/2502.18754 · HTML: https://mlsanity.com/paper/2502.18754/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Agentic recommender systems, one of 16 directions in LLM agents and tool use.
- Builds on 19 of its 21 references that are in the map.
- First posted on arXiv 26 Feb 2025.

## Placed in

- LLM agents and tool use / [Agentic recommender systems](https://mlsanity.com/agents/agentic-recommender-systems/)

## Builds on (most cited first, 19 of 19)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Neural Collaborative Filtering](https://mlsanity.com/paper/1708.05031/) (2017)
- [TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification](https://mlsanity.com/paper/2010.12421/) (2020)
- [A Survey on Large Language Models for Recommendation](https://mlsanity.com/paper/2305.19860/) (2023)
- [Towards Large Reasoning Models: A Survey of Reinforced Reasoning with Large Language Models](https://mlsanity.com/paper/2501.09686/) (2025)
- [Large Language Models Empowered Agent-based Modeling and Simulation: A Survey and Perspectives](https://mlsanity.com/paper/2312.11970/) (2023)
- [Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoders](https://mlsanity.com/paper/2403.03952/) (2024)
- [Understanding World or Predicting Future? A Comprehensive Survey of World Models](https://mlsanity.com/paper/2411.14499/) (2024)
- [AgentSquare: Automatic LLM Agent Search in Modular Design Space](https://mlsanity.com/paper/2410.06153/) (2024)
- [RecMind: Large Language Model Powered Agent For Recommendation](https://mlsanity.com/paper/2308.14296/) (2023)
- [AgentCF: Collaborative Learning with Autonomous Language Agents for Recommender Systems](https://mlsanity.com/paper/2310.09233/) (2023)
- [On Generative Agents in Recommendation](https://mlsanity.com/paper/2310.10108/) (2023)
- [Yelp Dataset Challenge: Review Rating Prediction](https://mlsanity.com/paper/1605.05362/) (2016)
- [LLaRA: Large Language-Recommendation Assistant](https://mlsanity.com/paper/2312.02445/) (2023)
- [Division-of-Thoughts: Harnessing Hybrid Language Model Synergy for Efficient On-Device Agents](https://mlsanity.com/paper/2502.04392/) (2025)
- [Large Language Model-driven Meta-structure Discovery in Heterogeneous Information Network](https://mlsanity.com/paper/2402.11518/) (2024)

## Built on by (most cited first, 12 of 12)

- [A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrieval](https://mlsanity.com/paper/2503.05659/) (2025)
- [AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems](https://mlsanity.com/paper/2505.19623/) (2025)
- [FingerTip 20K: A Benchmark for Proactive and Personalized Mobile LLM Agents](https://mlsanity.com/paper/2507.21071/) (2025)
- [AgentSwift: Efficient LLM Agent Design via Value-guided Hierarchical Search](https://mlsanity.com/paper/2506.06017/) (2025)
- [TAI3: Testing Agent Integrity in Interpreting User Intent](https://mlsanity.com/paper/2506.07524/) (2025)
- [BalDRO: A Distributionally Robust Optimization based Framework for Large Language Model Unlearning](https://mlsanity.com/paper/2601.09172/) (2026)
- [ScienceDB AI: An LLM-Driven Agentic Recommender System for Large-Scale Scientific Data Sharing Services](https://mlsanity.com/paper/2601.01118/) (2026)
- [CrimeMind: Simulating Urban Crime with Multi-Modal LLM Agents](https://mlsanity.com/paper/2506.05981/) (2025)
- [SOCIA-nabla: Textual Gradient Meets Multi-Agent Orchestration for Automated Simulator Generation](https://mlsanity.com/paper/2505.12006/) (2025)
- [SOCIA-Nabla: Textual Gradient Meets Multi-Agent Orchestration for Automated Simulator Generation](https://mlsanity.com/paper/2510.18551/) (2025)
- [SOCIA-EVO: Automated Simulator Construction via Dual-Anchored Bi-Level Optimization](https://mlsanity.com/paper/2604.17351/) (2026)
- [APeB: Benchmarking Personalization Ability of Large Language Model Agents](https://mlsanity.com/paper/2607.03162/) (2026)

## Abstract (arXiv)

The AgentSociety Challenge is the first competition in the Web Conference that aims to explore the potential of Large Language Model (LLM) agents in modeling user behavior and enhancing recommender systems on web platforms. The Challenge consists of two tracks: the User Modeling Track and the Recommendation Track. Participants are tasked to utilize a combined dataset from Yelp, Amazon, and Goodreads, along with an interactive environment simulator, to develop innovative LLM agents. The Challenge has attracted 295 teams across the globe and received over 1,400 submissions in total over the course of 37 official competition days. The participants have achieved 21.9% and 20.3% performance improvement for Track 1 and Track 2 in the Development Phase, and 9.1% and 15.9% in the Final Phase, representing a significant accomplishment. This paper discusses the detailed designs of the Challenge, analyzes the outcomes, and highlights the most successful LLM agent designs. To support further research and development, we have open-sourced the benchmark environment at https://tsinghua-fib-lab.github.io/AgentSocietyChallenge.