# CREW: Facilitating Human-AI Teaming Research

arXiv: https://arxiv.org/abs/2408.00170 · HTML: https://mlsanity.com/paper/2408.00170/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 74 references that are in the map.
- First posted on arXiv 31 Jul 2024.

## Builds on (most cited first, 30 of 33)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms](https://mlsanity.com/paper/1911.10635/) (2019)
- [Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning](https://mlsanity.com/paper/2107.09645/) (2021)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [Reward learning from human preferences and demonstrations in Atari](https://mlsanity.com/paper/1811.06521/) (2018)
- [StarCraft II: A New Challenge for Reinforcement Learning](https://mlsanity.com/paper/1708.04782/) (2017)
- [Unity: A General Platform for Intelligent Agents](https://mlsanity.com/paper/1809.02627/) (2018)
- [Semantic Photo Manipulation with a Generative Image Prior](https://mlsanity.com/paper/2005.07727/) (2020)
- [Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces](https://mlsanity.com/paper/1709.10163/) (2017)
- [Explainable Artificial Intelligence for Autonomous Driving: A Comprehensive Overview and Field Guide for Future Research Directions](https://mlsanity.com/paper/2112.11561/) (2021)
- [TorchRL: A data-driven decision-making library for PyTorch](https://mlsanity.com/paper/2306.00577/) (2023)
- [Deep Reinforcement Learning from Policy-Dependent Human Feedback](https://mlsanity.com/paper/1902.04257/) (2019)
- [Deriving Machine Attention from Human Rationales](https://mlsanity.com/paper/1808.09367/) (2018)
- [Perfection Not Required? Human-AI Partnerships in Code Translation](https://mlsanity.com/paper/2104.03820/) (2021)
- [DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback](https://mlsanity.com/paper/1810.11748/) (2018)
- [FIND: Human-in-the-Loop Debugging Deep Text Classifiers](https://mlsanity.com/paper/2010.04987/) (2020)
- [Human-AI Collaboration: The Effect of AI Delegation on Human Task Performance and Task Satisfaction](https://mlsanity.com/paper/2303.09224/) (2023)
- [Analysis of the hands in egocentric vision: A survey](https://mlsanity.com/paper/1912.10867/) (2019)
- [Uni-RLHF: Universal Platform and Benchmark Suite for Reinforcement Learning with Diverse Human Feedback](https://mlsanity.com/paper/2402.02423/) (2024)
- [Ancilia: Scalable Intelligent Video Surveillance for the Artificial Intelligence of Things](https://mlsanity.com/paper/2301.03561/) (2023)
- [FRESH: Interactive Reward Shaping in High-Dimensional State Spaces using Human Feedback](https://mlsanity.com/paper/2001.06781/) (2020)
- [RLHF-Blender: A Configurable Interactive Interface for Learning from Diverse Human Feedback](https://mlsanity.com/paper/2308.04332/) (2023)
- [Visual Hide and Seek](https://mlsanity.com/paper/1910.07882/) (2019)
- [DERAIL: Diagnostic Environments for Reward And Imitation Learning](https://mlsanity.com/paper/2012.01365/) (2020)
- [Visual Perspective Taking for Opponent Behavior Modeling](https://mlsanity.com/paper/2105.05145/) (2021)

## Built on by (most cited first, 11 of 11)

- [MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents](https://mlsanity.com/paper/2503.01935/) (2025)
- [A Comprehensive Survey on Multi-Agent Cooperative Decision-Making: Scenarios, Approaches, Challenges and Perspectives](https://mlsanity.com/paper/2503.13415/) (2025)
- [A Survey on Agent Workflow -- Status and Future](https://mlsanity.com/paper/2508.01186/) (2025)
- [SOP-Agent: Empower General Purpose AI Agent with Domain-Specific SOPs](https://mlsanity.com/paper/2501.09316/) (2025)
- [CREW-WILDFIRE: Benchmarking Agentic Multi-Agent Collaborations at Scale](https://mlsanity.com/paper/2507.05178/) (2025)
- [Through the Lens of Human-Human Collaboration: A Configurable Research Platform for Exploring Human-Agent Collaboration](https://mlsanity.com/paper/2509.18008/) (2025)
- [Enabling Multi-Robot Collaboration from Single-Human Guidance](https://mlsanity.com/paper/2409.19831/) (2024)
- [GUIDE: Real-Time Human-Shaped Agents](https://mlsanity.com/paper/2410.15181/) (2024)
- [Pref-GUIDE: Continual Policy Learning from Real-Time Human Feedback via Preference-Based Learning](https://mlsanity.com/paper/2508.07126/) (2025)
- [EpochX: Building the Infrastructure for an Emergent Agent Civilization](https://mlsanity.com/paper/2603.27304/) (2026)
- [Gricea: An Open Science Platform for Conversational AI Research](https://mlsanity.com/paper/2609.22039/) (2026)

## Abstract (arXiv)

With the increasing deployment of artificial intelligence (AI) technologies, the potential of humans working with AI agents has been growing at a great speed. Human-AI teaming is an important paradigm for studying various aspects when humans and AI agents work together. The unique aspect of Human-AI teaming research is the need to jointly study humans and AI agents, demanding multidisciplinary research efforts from machine learning to human-computer interaction, robotics, cognitive science, neuroscience, psychology, social science, and complex systems. However, existing platforms for Human-AI teaming research are limited, often supporting oversimplified scenarios and a single task, or specifically focusing on either human-teaming research or multi-agent AI algorithms. We introduce CREW, a platform to facilitate Human-AI teaming research in real-time decision-making scenarios and engage collaborations from multiple scientific disciplines, with a strong emphasis on human involvement. It includes pre-built tasks for cognitive studies and Human-AI teaming with expandable potentials from our modular design. Following conventional cognitive neuroscience research, CREW also supports multimodal human physiological signal recording for behavior analysis. Moreover, CREW benchmarks real-time human-guided reinforcement learning agents using state-of-the-art algorithms and well-tuned baselines. With CREW, we were able to conduct 50 human subject studies within a week to verify the effectiveness of our benchmark.