mlsanityData to 7 Oct 2026

AI-Gadget Kit: Integrating Swarm User Interfaces with LLM-driven Agents for Rich Tabletop Game Applications

arXiv 2407.17086 ↗First posted 24 Jul 2024cs.HCcited by 0

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

While Swarm User Interfaces (SUIs) have succeeded in enriching tangible interaction experiences, their limitations in autonomous action planning have hindered the potential for personalized and dynamic interaction generation in tabletop games. Based on the AI-Gadget Kit we developed, this paper explores how to integrate LLM-driven agents within tabletop games to enable SUIs to execute complex interaction tasks. After defining the design space of this kit, we elucidate the method for designing agents that can extend the meta-actions of SUIs to complex motion planning. Furthermore, we introduce an add-on prompt method that simplifies the design process for four interaction behaviors and four interaction relationships in tabletop games. Lastly, we present several application scenarios that illustrate the potential of AI-Gadget Kit to construct personalized interaction in SUI tabletop games. We expect to use our work as a case study to inspire research on multi-agent-driven SUI for other scenarios with complex interaction tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 13 of its 37 references that are in the map. First posted on arXiv 24 Jul 2024.

Builds on

13 of 37 references.

References · 13

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  3. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  4. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  5. 2023Discuss Before Moving: Visual Language Navigation via Multi-expert Discussionscited by 108
  6. 2019Look, Listen, and Act: Towards Audio-Visual Embodied Navigationcited by 95
  7. 2020Environment-agnostic Multitask Learning for Natural Language Grounded Navigationcited by 43
  8. 2023March in Chat: Interactive Prompting for Remote Embodied Referring Expressioncited by 34
  9. 2020RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketchingcited by 14
  10. 2021Player-AI Interaction: What Neural Network Games Reveal About AI as Playcited by 8
  11. 2022Sketched Reality: Sketching Bi-Directional Interactions Between Virtual and Physical Worlds with AR and Actuated Tangible UIcited by 5
  12. 2019ShapeBots: Shape-changing Swarm Robotscited by 3
1 more
  1. 2023HoloBots: Augmenting Holographic Telepresence with Mobile Robots for Tangible Remote Collaboration in Mixed Realitycited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Static Vs. Agentic Game Master AI for Facilitating Solo Role-Playing Experiences
  2. 2023LLM-Based Agent Society Investigation: Collaboration and Confrontation in Avalon Gameplay
  3. 2025IndoorWorld: Integrating Physical Task Solving and Social Simulation in A Heterogeneous Multi-Agent Environment
  4. 2025Collaborating Action by Action: A Multi-agent LLM Framework for Embodied Reasoning
  5. 2026GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents
  6. 2025General Modular Harness for LLM Agents in Multi-Turn Gaming Environments
  7. 2023Tachikuma: Understading Complex Interactions with Multi-Character and Novel Objects by Large Language Models
  8. 2025SimWorld: An Open-ended Realistic Simulator for Autonomous Agents in Physical and Social Worlds