# HARMONIC: Cognitive and Control Collaboration in Human-Robotic Teams

arXiv: https://arxiv.org/abs/2409.18047 · HTML: https://mlsanity.com/paper/2409.18047/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 49 references that are in the map.
- First posted on arXiv 26 Sep 2024.

## Builds on (most cited first, 13 of 13)

- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://mlsanity.com/paper/2310.08864/) (2023)
- [Generalized Planning in PDDL Domains with Pretrained Large Language Models](https://mlsanity.com/paper/2305.11014/) (2023)
- [Behavior Trees in Robotics and AI: An Introduction](https://mlsanity.com/paper/1709.00084/) (2017)
- [Plug in the Safety Chip: Enforcing Constraints for LLM-driven Robot Agents](https://mlsanity.com/paper/2309.09919/) (2023)
- [Grounding LLMs For Robot Task Planning Using Closed-loop State Feedback](https://mlsanity.com/paper/2402.08546/) (2024)
- [Practical Verification of Decision-Making in Agent-Based Autonomous Systems](https://mlsanity.com/paper/1310.2431/) (2013)
- [On the Implementation of Behavior Trees in Robotics](https://mlsanity.com/paper/2106.15227/) (2021)
- [Planning for Decentralized Control of Multiple Robots Under Uncertainty](https://mlsanity.com/paper/1402.2871/) (2014)
- [KT-BT: A Framework for Knowledge Transfer Through Behavior Trees in Multi-Robot Systems](https://mlsanity.com/paper/2209.02886/) (2022)
- [Impact of Heterogeneity in Multi-Robot Systems on Collective Behaviors Studied Using a Search and Rescue Problem](https://mlsanity.com/paper/2012.14329/) (2020)
- [Heterogeneous Full-body Control of a Mobile Manipulator with Behavior Trees](https://mlsanity.com/paper/2210.08600/) (2022)
- [IKT-BT: Indirect Knowledge Transfer Behavior Tree Framework for Multi-Robot Systems Through Communication Eavesdropping](https://mlsanity.com/paper/2312.11802/) (2023)

## Built on by (most cited first, 2 of 2)

- [Gricean Norms as a Basis for Effective Collaboration](https://mlsanity.com/paper/2503.14484/) (2025)
- [Shapes of Cognition for Computational Cognitive Modeling](https://mlsanity.com/paper/2509.13288/) (2025)

## Abstract (arXiv)

This paper describes HARMONIC, a cognitive-robotic architecture that integrates the OntoAgent cognitive framework with general-purpose robot control systems applied to human-robot teaming (HRT). HARMONIC incorporates metacognition, meaningful natural language communication, and explainability capabilities required for developing mutual trust in HRT. Through simulation experiments involving a joint search task performed by a heterogeneous team of two HARMONIC-based robots and a human operator, we demonstrate heterogeneous robots that coordinate their actions, adapt to complex scenarios, and engage in natural human-robot communication. Evaluation results show that HARMONIC-based robots can reason about plans, goals, and team member attitudes while providing clear explanations for their decisions, which are essential requirements for realistic human-robot teaming.