mlsanityData to 14 Oct 2026

HoosierHelp: Benchmarking LLM Agents for Social Service Navigation

arXiv 2608.09946 ↗First posted 3 Jul 2026cs.HC cs.AIcited by 0

On the map: LLM agents and tool use → Social simulation and multi-agent debate

Abstract (arXiv)

Social service navigation requires connecting help-seeking individuals to resources that satisfy their needs and specific constraints. Although LLM agents offer a promising interface for conversational resource navigation, existing benchmarks do not capture the interaction complexity and constraint-grounding demands of this setting. We introduce HoosierHelp, an interactive benchmark grounded in 3,971 Indiana public social service resources. Agents interact with simulated users, issue structured resource-search calls, handle non-ideal interactions, and select the final resources returned by the tool. HoosierHelp enhances the realism of simulated users by varying their need structure, constraint satisfiability, and behavior patterns, including impatience, rambling, unsupported requests, and self-contradiction. Experiments on 240 samples across seven LLMs show that current LLM agents remain substantially unreliable for social service navigation. Performance drops sharply on fallback-required and self-contradictory conversations, highlighting the need for agents that are more robust to complex and non-ideal user interactions.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 22 of its 91 references that are in the map. First posted on arXiv 3 Jul 2026.

Builds on

22 of 91 references.

References · 22

  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  2. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  3. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  4. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
  5. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 1,040
  6. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 591
  7. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 295
  8. 2025The Obvious Invisible Threat: LLM-Powered GUI Agents' Vulnerability to Fine-Print Injectionscited by 59
  9. 2024CLEAR: Towards Contextual LLM-Empowered Privacy Policy Analysis and Risk Generation for Large Language Model Applicationscited by 33
  10. 2025AgentRouter: A Knowledge-Graph-Guided LLM Router for Collaborative Multi-Agent Question Answeringcited by 32
  11. 2023GraphPatcher: Mitigating Degree Bias for Graph Neural Networks via Test-time Augmentationcited by 31
  12. 2022INSCIT: Information-Seeking Conversations with Mixed-Initiative Interactionscited by 27
10 more
  1. 2025AutoData: A Multi-Agent System for Open Web Data Collectioncited by 25
  2. 2025CONFETTI: Conversational Function-Calling Evaluation Through Turn-Level Interactionscited by 18
  3. 2025Non-Collaborative User Simulators for Tool Agentscited by 18
  4. 2025MAPRO: Recasting Multi-Agent Prompt Optimization as Maximum a Posteriori Inferencecited by 17
  5. 2024NGQA: A Nutritional Graph Question Answering Benchmark for Personalized Health-aware Nutritional Reasoningcited by 16
  6. 2026EvolveRouter: Co-Evolving Routing and Prompt for Multi-Agent Question Answeringcited by 15
  7. 2025LLM-Empowered Class Imbalanced Graph Prompt Learning for Online Drug Trafficking Detectioncited by 11
  8. 2025NG-Router: Graph-Supervised Multi-Agent Collaboration for Nutrition Question Answeringcited by 10
  9. 2026Drift-Bench: Diagnosing Cooperative Breakdowns in LLM Agents under Input Faults via Multi-Turn Interactioncited by 9
  10. 2026GLEN-Bench: A Graph-Language based Benchmark for Nutritional Healthcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Verifiable Social Reasoning for LLM Assistants
  2. 2025How Social is It? A Benchmark for LLMs' Capabilities in Multi-user Multi-turn Social Agent Tasks
  3. 2026CRAB-Bench: Evaluating LLM Agents under Complex Task Dependencies and Human-aligned User Simulation
  4. 2026ComPASS: Towards Personalized Agentic Social Support via Tool-Augmented Companionship
  5. 2025SI-Bench: Benchmarking Social Intelligence of Large Language Models in Human-to-Human Conversations
  6. 2025SOTOPIA-S4: a user-friendly system for flexible, customizable, and large-scale social simulation
  7. 2025The Social Laboratory: A Psychometric Framework for Multi-Agent LLM Evaluation
  8. 2026SAGE: A Service Agent Graph-guided Evaluation Benchmark