mlsanityData to 14 Oct 2026

Through the Lens of Human-Human Collaboration: A Configurable Research Platform for Exploring Human-Agent Collaboration

arXiv 2509.18008 ↗First posted 22 Sep 2025cs.HC cs.AI cs.CLcited by 4

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Intelligent systems have traditionally been designed as tools rather than collaborators, often lacking critical characteristics that collaboration partnerships require. Recent advances in large language model (LLM) agents open new opportunities for human-LLM-agent collaboration by enabling natural communication and various social and cognitive behaviors. Yet it remains unclear whether principles of computer-mediated collaboration established in HCI and CSCW persist, change, or fail when humans collaborate with LLM agents. To support systematic investigations of these questions, we introduce an open and configurable research platform for HCI researchers. The platform's modular design allows seamless adaptation of classic CSCW experiments and manipulation of theory-grounded interaction controls. We demonstrate the platform's research efficacy and usability through three case studies: (1) two Shape Factory experiments for resource negotiation with 16 participants, (2) one Hidden Profile experiment for information pooling with 16 participants, and (3) a participatory cognitive walkthrough with five HCI researchers to refine workflows of researcher interface for experiment setup and analysis.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 37 of its 121 references that are in the map. First posted on arXiv 22 Sep 2025.

Builds on

37 of 121 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Multi-Agent-as-Judge: Aligning LLM-Agent-Based Automated Evaluation with Multi-Dimensional Human Evaluation28 Jul
  2. 2024Agent-as-a-Judge: Evaluate Agents with Agents14 Oct
  3. 2023AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors21 Aug
  4. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society31 Mar

Other references · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  7. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  8. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  9. 2024A Survey on LLM-as-a-Judgecited by 1,423
  10. 2023ChatDev: Communicative Agents for Software Developmentcited by 919
  11. 2022Out of One, Many: Using Language Models to Simulate Human Samplescited by 752
  12. 2022Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studiescited by 515
21 more
  1. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 349
  2. 2023Role-Play with Large Language Modelscited by 294
  3. 2020Questioning the AI: Informing Design Practices for Explainable AI User Experiencescited by 253
  4. 2022CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilitiescited by 249
  5. 2024LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individualscited by 249
  6. 2025AgentSociety: Large-Scale Simulation of LLM-Driven Generative Agents Advances Understanding of Human Behaviors and Societycited by 186
  7. 2024When combinations of humans and AI are useful: A systematic review and meta-analysiscited by 106
  8. 2021Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Potcited by 99
  9. 2019Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AIcited by 79
  10. 2022Human-AI Collaboration Enables More Empathic Conversations in Text-based Peer-to-Peer Mental Health Supportcited by 68
  11. 2025UXAgent: An LLM Agent-Based Usability Testing Framework for Web Designcited by 40
  12. 2023Rethinking Human-AI Collaboration in Complex Medical Decision Making: A Case Study in Sepsis Diagnosiscited by 30
  13. 2025OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulationcited by 28
  14. 2025Towards a Design Guideline for RPA Evaluation: A Survey of Large Language Model-Based Role-Playing Agentscited by 22
  15. 2020Empirica: a virtual lab for high-throughput macro-level experimentscited by 16
  16. 2025Collaborating with AI Agents: Field Experiments on Teamwork, Productivity, and Performancecited by 14
  17. 2024Investigating Interaction Modes and User Agency in Human-LLM Collaboration for Domain-Specific Data Analysiscited by 12
  18. 2023Competent but Rigid: Identifying the Gap in Empowering AI to Participate Equally in Group Decision-Makingcited by 11
  19. 2024CREW: Facilitating Human-AI Teaming Researchcited by 11
  20. 2025Toward a Human-Centered Evaluation Framework for Trustworthy LLM-Powered GUI Agentscited by 11
  21. 2025VirtLab: An AI-Powered System for Flexible, Customizable, and Large-scale Team Simulationscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Comparing Human Oversight Strategies for Computer-Use Agentscited by 6
  2. 2026CollabSkill: Evaluating Human-Agent Collaboration On Real-World Taskscited by 1
  3. 2026Humans' ALMANAC: A Human Collaboration Dataset of Action-Level Mental Model Annotations for Agent Collaborationcited by 0
  4. 2026Gricea: An Open Science Platform for Conversational AI Researchcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From Human-Human Collaboration to Human-Agent Collaboration: A Vision, Design Philosophy, and an Empirical Framework for Achieving Successful Partnerships Between Humans and LLM Agents
  2. 2025LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey
  3. 2023Exploring Collaboration Mechanisms for LLM Agents: A Social Psychology View
  4. 2025A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomy
  5. 2026HAS-Bench: Evaluating LLM-Based Human-Agent Systems under Configurable Human Participation
  6. 2026Toward Human-Centered Multi-Agent Systems: Integrating Cognition, Culture, Values, and Cooperation in AI Agents
  7. 2024Your Co-Workers Matter: Evaluating Collaborative Capabilities of Language Models in Blocks World
  8. 2024Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration