mlsanityData to 14 Oct 2026

Shopping MMLU: A Massive Multi-Task Online Shopping Benchmark for Large Language Models

arXiv 2410.20745 ↗First posted 28 Oct 2024cs.LG cs.AIcited by 20

Not on the map. Why?

Abstract (arXiv)

Online shopping is a complex multi-task, few-shot learning problem with a wide and evolving range of entities, relations, and tasks. However, existing models and benchmarks are commonly tailored to specific tasks, falling short of capturing the full complexity of online shopping. Large Language Models (LLMs), with their multi-task and few-shot learning abilities, have the potential to profoundly transform online shopping by alleviating task-specific engineering efforts and by providing users with interactive conversations. Despite the potential, LLMs face unique challenges in online shopping, such as domain-specific concepts, implicit knowledge, and heterogeneous user behaviors. Motivated by the potential and challenges, we propose Shopping MMLU, a diverse multi-task online shopping benchmark derived from real-world Amazon data. Shopping MMLU consists of 57 tasks covering 4 major shopping skills: concept understanding, knowledge reasoning, user behavior alignment, and multi-linguality, and can thus comprehensively evaluate the abilities of LLMs as general shop assistants. With Shopping MMLU, we benchmark over 20 existing LLMs and uncover valuable insights about practices and prospects of building versatile LLM-based shop assistants. Shopping MMLU can be publicly accessed at https://github.com/KL4805/ShoppingMMLU. In addition, with Shopping MMLU, we host a competition in KDD Cup 2024 with over 500 participating teams. The winning solutions and the associated workshop can be accessed at our website https://amazon-kddcup24.github.io/.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Builds on 37 of its 58 references that are in the map. First posted on arXiv 28 Oct 2024.

Builds on

37 of 58 references.

References · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2023Qwen Technical Reportcited by 3,344
  12. 2023Mistral 7Bcited by 3,164
25 more
  1. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  2. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  3. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  4. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  5. 2017A Survey on Multi-Task Learningcited by 840
  6. 2018Session-based Recommendation with Graph Neural Networkscited by 312
  7. 2023Large Language Models are Zero-Shot Rankers for Recommender Systemscited by 309
  8. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
  9. 2023Can Large Language Models Transform Computational Social Science?cited by 250
  10. 2024Bridging Language and Items for Retrieval and Recommendation: Benchmarking LLMs as Semantic Encoderscited by 224
  11. 2023LLMRec: Large Language Models with Graph Augmentation for Recommendationcited by 172
  12. 2023GPT-NER: Named Entity Recognition via Large Language Modelscited by 160
  13. 2023Text Is All You Need: Learning Language Representations for Sequential Recommendationcited by 149
  14. 2022Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Searchcited by 65
  15. 2018OpenTag: Open Attribute Value Extraction from Product Profiles [Deep Learning, Active Learning, Named Entity Recognition]cited by 63
  16. 2021MAVE: A Product Dataset for Multi-source Attribute Value Extractioncited by 53
  17. 2023EcomGPT: Instruction-tuning Large Language Models with Chain-of-Task Tasks for E-commercecited by 49
  18. 2024eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Datacited by 46
  19. 2019AmazonQA: A Review-Based Question Answering Taskcited by 43
  20. 2023Amazon-M2: A Multilingual Multi-locale Shopping Session Dataset for Recommendation and Text Generationcited by 42
  21. 2022Retrieval-Augmented Multilingual Keyphrase Generation with Retriever-Generator Iterative Trainingcited by 16
  22. 2021Design and Operation of Shared Machine Learning Clusters on Campuscited by 12
  23. 2023Enhancing User Intent Capture in Session-Based Recommendation with Attribute Patternscited by 11
  24. 2022Short Text Pre-training with Extended Token Classification for E-commerce Query Understandingcited by 5
  25. 2022Some Practice for Improving the Search Results of E-commercecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 6.

20 papers.

Sort:

By area:

  1. 2025Agentic Large Language Models, a surveycited by 108
  2. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  3. 2025Generative to Agentic AI: Survey, Conceptualization, and Challengescited by 33
  4. 2025Fraud-R1 : A Multi-Round Benchmark for Assessing the Robustness of LLM Against Augmented Fraud and Phishing Inducementscited by 32
  5. 2025ShoppingBench: A Real-World Intent-Grounded Shopping Benchmark for LLM-based Agentscited by 28
  6. 2025Optimas: Optimizing Compound AI Systems with Globally Aligned Local Rewardscited by 24
  7. 2025What Is Your AI Agent Buying? Evaluation, Biases, Model Dependence, & Emerging Implications for Agentic E-Commercecited by 22
  8. 2025A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experimentscited by 12
  9. 2025ShoppingComp: Are LLMs Really Ready for Your Shopping Cart?cited by 9
  10. 2026AgenticShop: Benchmarking Agentic Product Curation for Personalized Web Shoppingcited by 7
  11. 2025GuessArena: Guess Who I Am? A Self-Adaptive Framework for Evaluating LLMs in Domain-Specific Knowledge and Reasoningcited by 3
  12. 2025Compass-V2 Technical Reportcited by 2
8 more
  1. 2025Prompt and Parameter Co-Optimization for Large Language Modelscited by 2
  2. 2026Shopping Reasoning Bench: An Expert-Authored Benchmark for Multi-Turn Conversational Shopping Assistantscited by 2
  3. 2025Towards Reliable Evaluation of Large Language Models for Multilingual and Multimodal E-Commerce Applicationscited by 1
  4. 2026Interactive Task Alignment as a POMDPcited by 1
  5. 2025AI Playing Business Games: Benchmarking Large Language Models on Managerial Decision-Making in Dynamic Simulationscited by 0
  6. 2026Hidden Ads: Behavior Triggered Semantic Backdoors for Advertisement Injection in Vision Language Modelscited by 0
  7. 2026Theoretical Limits of Language Model Alignmentcited by 0
  8. 2026RecoAtlas: From Semantic Plausibility to Set-Level Utility in LLM Recommendation Agentscited by 0