# Towards Actionable Surgical Team Dynamics: from Teamwork to Counterfactual Annotations

arXiv: https://arxiv.org/abs/2608.23344 · HTML: https://mlsanity.com/paper/2608.23344/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 61 references that are in the map.
- First posted on arXiv 24 Aug 2026.

## Builds on (most cited first, 15 of 15)

- [Explanation in Artificial Intelligence: Insights from the Social Sciences](https://mlsanity.com/paper/1706.07269/) (2017)
- [Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR](https://mlsanity.com/paper/1711.00399/) (2017)
- [EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos](https://mlsanity.com/paper/1602.03012/) (2016)
- [Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos](https://mlsanity.com/paper/2109.03223/) (2021)
- [A survey of algorithmic recourse: definitions, formulations, solutions, and prospects](https://mlsanity.com/paper/2010.04050/) (2020)
- [Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Triplets](https://mlsanity.com/paper/2007.05405/) (2020)
- [MM-OR: A Large Multimodal Operating Room Dataset for Semantic Understanding of High-Intensity Surgical Environments](https://mlsanity.com/paper/2503.02579/) (2025)
- [SALSA: A Novel Dataset for Multimodal Group Behavior Analysis](https://mlsanity.com/paper/1506.06882/) (2015)
- [Detecting Low Rapport During Natural Interactions in Small Groups from Non-Verbal Behaviour](https://mlsanity.com/paper/1801.06055/) (2018)
- [Towards an AI Coach to Infer Team Mental Model Alignment in Healthcare](https://mlsanity.com/paper/2102.08507/) (2021)
- [Putting Context in Context: the Impact of Discussion Structure on Text Classification](https://mlsanity.com/paper/2402.02975/) (2024)
- [Boosting Team Modeling through Tempo-Relational Representation Learning](https://mlsanity.com/paper/2507.13305/) (2025)
- [PoCaP Corpus: A Multimodal Dataset for Smart Operating Room Speech Assistant using Interventional Radiology Workflow Analysis](https://mlsanity.com/paper/2206.12320/) (2022)
- [Actionable Real-Time Modeling of Surgical Team Dynamics via Time-Expanded Interaction Graphs](https://mlsanity.com/paper/2605.04169/) (2026)
- [CliniDial: A Naturally Occurring Multimodal Dialogue Dataset for Team Reflection in Action During Clinical Operation](https://mlsanity.com/paper/2506.12936/) (2025)

## Abstract (arXiv)

Modeling team interactions in high-stakes environments such as operating rooms is critical for understanding how coordination, communication, and individual behaviors shape team performance and safety outcomes. Existing datasets in this domain are often fragmented across modalities, annotation schemes, and formats, limiting their ability to support integrated analyses of real-world collaborative processes. We address this limitation by introducing an extended multimodal dataset for surgical team interaction analysis, built from real operating room recordings. Starting from an existing corpus, we construct an analysis-ready version of the data by providing speaker diarization, transcripts, and multi-level annotations capturing team performance, interaction processes, and individual characteristics. Team performance is assessed using a standardized surgical teamwork evaluation protocol, while interaction quality and individual attributes are annotated through structured rating schemes covering collaboration, group dynamics, and non-technical skills. To further support the study of coordination breakdowns and performance variability, we introduce counterfactual annotations that describe plausible alternative team outcomes in the presence of observed interaction failures, enabling analysis of how specific behavioral patterns may relate to different trajectories of team performance. In addition, we provide structured temporal and relational representations designed to support computational modeling of teamwork processes and the design of AI-assisted collaborative systems. The dataset is designed to support the study of how individual actions, interaction patterns, and team-level processes jointly contribute to team outcomes in surgical settings, providing a unified resource for analyzing collaborative behavior in high-stakes domains.