mlsanityData to 14 Oct 2026

Auto-SLURP: A Benchmark Dataset for Evaluating Multi-Agent Frameworks in Smart Personal Assistant

arXiv 2504.18373 ↗First posted 25 Apr 2025cs.CLcited by 1

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

In recent years, multi-agent frameworks powered by large language models (LLMs) have advanced rapidly. Despite this progress, there is still a notable absence of benchmark datasets specifically tailored to evaluate their performance. To bridge this gap, we introduce Auto-SLURP, a benchmark dataset aimed at evaluating LLM-based multi-agent frameworks in the context of intelligent personal assistants. Auto-SLURP extends the original SLURP dataset -- initially developed for natural language understanding tasks -- by relabeling the data and integrating simulated servers and external services. This enhancement enables a comprehensive end-to-end evaluation pipeline, covering language understanding, task execution, and response generation. Our experiments demonstrate that Auto-SLURP presents a significant challenge for current state-of-the-art frameworks, highlighting that truly reliable and intelligent multi-agent personal assistants remain a work in progress. The dataset and related code are available at https://github.com/lorashen/Auto-SLURP/.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 23 of its 43 references that are in the map. First posted on arXiv 25 Apr 2025.

Builds on

23 of 43 references.

References · 23

  1. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
  2. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  3. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  4. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  5. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 367
  6. 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agentscited by 339
  7. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 274
  8. 2023AutoAgents: A Framework for Automatic Agent Generationcited by 216
  9. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 198
  10. 2019Benchmarking Natural Language Understanding Services for building Conversational Agentscited by 166
  11. 2020Watch-And-Help: A Challenge for Social Perception and Human-AI Collaborationcited by 133
  12. 2023T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Stepcited by 93
11 more
  1. 2023MindAgent: Emergent Gaming Interactioncited by 88
  2. 2021The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AIcited by 65
  3. 2024LegalAgentBench: Evaluating LLM Agents in Legal Domaincited by 59
  4. 2024AgentLite: A Lightweight Library for Building and Advancing Task-Oriented LLM Agent Systemcited by 53
  5. 2024CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agentscited by 42
  6. 2024AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarioscited by 40
  7. 2024ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarioscited by 35
  8. 2024Unraveling the Mystery of Scaling Laws: Part Icited by 24
  9. 2022Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Surveycited by 14
  10. 2016End-to-End Joint Learning of Natural Language Understanding and Dialogue Managercited by 6
  11. 2021Knowledge-enhanced Session-based Recommendation with Temporal Transformercited by 2

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Cochain: Balancing Insufficient and Excessive Collaboration in LLM Agent Workflowscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026π-Bench: Evaluating Proactive Personal Assistant Agents in Long-Horizon Workflows
  2. 2025PersonaLens: A Benchmark for Personalization Evaluation in Conversational AI Assistants
  3. 2024CAMPHOR: Collaborative Agents for Multi-input Planning and High-Order Reasoning On Device
  4. 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation
  5. 2026Claw-Anything: Benchmarking Always-On Personal Assistants with Broader Access to User's Digital World
  6. 2026SMH-Bench: Benchmarking LLM Agents for Environment-Grounded Reasoning and Action in Smart Homes
  7. 2024Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents
  8. 2026Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis