# The CASTLE 2024 Dataset: Advancing the Art of Multimodal Understanding

arXiv: https://arxiv.org/abs/2503.17116 · HTML: https://mlsanity.com/paper/2503.17116/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 14 references that are in the map.
- First posted on arXiv 21 Mar 2025.

## Builds on (most cited first, 6 of 6)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [Rescaling Egocentric Vision](https://mlsanity.com/paper/2006.13256/) (2020)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [MEVA: A Large-Scale Multiview, Multimodal Video Dataset for Activity Detection](https://mlsanity.com/paper/2012.00914/) (2020)
- [EgoSim: An Egocentric Multi-view Simulator and Real Dataset for Body-worn Cameras during Motion and Activity](https://mlsanity.com/paper/2502.18373/) (2025)

## Built on by (most cited first, 16 of 16)

- [Ego-Grounding for Personalized Question-Answering in Egocentric Videos](https://mlsanity.com/paper/2604.01966/) (2026)
- [FIGROTD: A Friendly-to-Handle Dataset for Image Guided Retrieval with Optional Text](https://mlsanity.com/paper/2511.22247/) (2025)
- [MARS: Technical Report for the CASTLE Challenge at EgoVis 2026](https://mlsanity.com/paper/2605.18176/) (2026)
- [CuriosAI Submission to the CASTLE Challenge at EgoVis 2026](https://mlsanity.com/paper/2605.27800/) (2026)
- [CASTLE2026 Team WDL Technical Report](https://mlsanity.com/paper/2606.00712/) (2026)
- [3rd Place at CVPR 2026 CASTLE Challenge: Agentic Multi-View Long-Context Video Understanding via Hierarchical Knowledge Graph Retrieval](https://mlsanity.com/paper/2606.01933/) (2026)
- [Logi-PAR: Logic-Infused Patient Activity Recognition via Differentiable Rule](https://mlsanity.com/paper/2603.05184/) (2026)
- [Bringing a Personal Point of View: Evaluating Dynamic 3D Gaussian Splatting for Egocentric Scene Reconstruction](https://mlsanity.com/paper/2604.23803/) (2026)
- [Imprint: Online Memory Compression for Long-Horizon Egocentric QA](https://mlsanity.com/paper/2607.00696/) (2026)
- [OmniAssistBench: Assistant-style Interaction Benchmark for Omni-LLMs](https://mlsanity.com/paper/2608.21360/) (2026)
- [Results of the 1st Asynchronous CASTLE Challenge at the Joint Egocentric Vision Workshop in Conjunction with CVPR 2026](https://mlsanity.com/paper/2608.22914/) (2026)
- [Measuring the Behavioral Fidelity of Long-Horizon Human Activity Simulations](https://mlsanity.com/paper/2609.01257/) (2026)
- [Talking to Me or Someone Else? Rethinking Talk-to-Me Detection in Egocentric Videos](https://mlsanity.com/paper/2609.14118/) (2026)
- [CapMem: A Benchmark for Caption-Based Episodic Memory in Egocentric Video](https://mlsanity.com/paper/2609.17688/) (2026)
- [NextMe-800: Anticipating Personal Behavior from Months of Egocentric Video](https://mlsanity.com/paper/2610.01461/) (2026)
- [GesBench: When and Where Do Multimodal Language Models Perceive Social Meaning?](https://mlsanity.com/paper/2610.15129/) (2026)

## Abstract (arXiv)

Egocentric video has seen increased interest in recent years, as it is used in a range of areas. However, most existing datasets are limited to a single perspective. In this paper, we present the CASTLE 2024 dataset, a multimodal collection containing ego- and exo-centric (i.e., first- and third-person perspective) video and audio from 15 time-aligned sources, as well as other sensor streams and auxiliary data. The dataset was recorded by volunteer participants over four days in a fixed location and includes the point of view of 10 participants, with an additional 5 fixed cameras providing an exocentric perspective. The entire dataset contains over 600 hours of UHD video recorded at 50 frames per second. In contrast to other datasets, CASTLE 2024 does not contain any partial censoring, such as blurred faces or distorted audio. The dataset is available via https://castle-dataset.github.io/.