# Hierarchical and Multimodal Data for Daily Activity Understanding

arXiv: https://arxiv.org/abs/2504.17696 · HTML: https://mlsanity.com/paper/2504.17696/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 87 references that are in the map.
- First posted on arXiv 24 Apr 2025.

## Builds on (most cited first, 30 of 46)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Ego4D: Around the World in 3,000 Hours of Egocentric Video](https://mlsanity.com/paper/2110.07058/) (2021)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [Video Swin Transformer](https://mlsanity.com/paper/2106.13230/) (2021)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [MViTv2: Improved Multiscale Vision Transformers for Classification and Detection](https://mlsanity.com/paper/2112.01526/) (2021)
- [Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives](https://mlsanity.com/paper/2311.18259/) (2023)
- [Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities](https://mlsanity.com/paper/2203.14712/) (2022)
- [FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding](https://mlsanity.com/paper/2004.06704/) (2020)
- [SportsMOT: A Large Multi-Object Tracking Dataset in Multiple Sports Scenes](https://mlsanity.com/paper/2304.05170/) (2023)
- [Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models](https://mlsanity.com/paper/2301.06267/) (2023)
- [My View is the Best View: Procedure Learning from Egocentric Videos](https://mlsanity.com/paper/2207.10883/) (2022)
- [Future Transformer for Long-term Action Anticipation](https://mlsanity.com/paper/2205.14022/) (2022)
- [ViewFool: Evaluating the Robustness of Visual Recognition to Adversarial Viewpoints](https://mlsanity.com/paper/2210.03895/) (2022)
- [Long-Term Anticipation of Activities with Cycle Consistency](https://mlsanity.com/paper/2009.01142/) (2020)
- [CURE-TSR: Challenging Unreal and Real Environments for Traffic Sign Recognition](https://mlsanity.com/paper/1712.02463/) (2017)
- [Coarse to Fine Multi-Resolution Temporal Convolutional Network](https://mlsanity.com/paper/2105.10859/) (2021)
- [Contrastive Explanations in Neural Networks](https://mlsanity.com/paper/2008.00178/) (2020)
- [OLIVES Dataset: Ophthalmic Labels for Investigating Visual Eye Semantics](https://mlsanity.com/paper/2209.11195/) (2022)
- [C2F-TCN: A Framework for Semi and Fully Supervised Temporal Action Segmentation](https://mlsanity.com/paper/2212.11078/) (2022)
- [A Survey of Human Activity Recognition in Smart Homes Based on IoT Sensors Algorithms: Taxonomies, Challenges, and Opportunities with Deep Learning](https://mlsanity.com/paper/2111.04418/) (2021)
- [Introspective Learning : A Two-Stage Approach for Inference in Neural Networks](https://mlsanity.com/paper/2209.08425/) (2022)
- [TS-LSTM and Temporal-Inception: Exploiting Spatiotemporal Dynamics for Activity Recognition](https://mlsanity.com/paper/1703.10667/) (2017)
- [Geometric Features Informed Multi-person Human-object Interaction Recognition in Videos](https://mlsanity.com/paper/2207.09425/) (2022)
- [A Large-scale Varying-view RGB-D Action Dataset for Arbitrary-view Human Action Recognition](https://mlsanity.com/paper/1904.10681/) (2019)
- [Traffic Sign Detection under Challenging Conditions: A Deeper Look Into Performance Variations and Spectral Characteristics](https://mlsanity.com/paper/1908.11262/) (2019)
- [Clinically Labeled Contrastive Learning for OCT Biomarker Classification](https://mlsanity.com/paper/2305.15154/) (2023)

## Built on by (most cited first, 7 of 7)

- [Subject Invariant Contrastive Learning for Human Activity Recognition](https://mlsanity.com/paper/2507.03250/) (2025)
- [Multi-level and Multi-modal Action Anticipation](https://mlsanity.com/paper/2506.02382/) (2025)
- [Countering Multi-modal Representation Collapse through Rank-targeted Fusion](https://mlsanity.com/paper/2511.06450/) (2025)
- [Information Router for Mitigating Modality Dominance in Vision-Language Models](https://mlsanity.com/paper/2604.16264/) (2026)
- [MER-DG: Modality-Entropy Regularization for Multimodal Domain Generalization](https://mlsanity.com/paper/2605.01967/) (2026)
- [SynCo: Learning Cross-Modal Synergy by Contrasting Interaction Residuals](https://mlsanity.com/paper/2609.32846/) (2026)
- [GeoPID: Decomposing and Steering Visual Information in Vision-Language Models](https://mlsanity.com/paper/2610.08401/) (2026)

## Abstract (arXiv)

Daily Activity Recordings for Artificial Intelligence (DARai, pronounced "Dahr-ree") is a multimodal, hierarchically annotated dataset constructed to understand human activities in real-world settings. DARai consists of continuous scripted and unscripted recordings of 50 participants in 10 different environments, totaling over 200 hours of data from 20 sensors including multiple camera views, depth and radar sensors, wearable inertial measurement units (IMUs), electromyography (EMG), insole pressure sensors, biomonitor sensors, and gaze tracker. To capture the complexity in human activities, DARai is annotated at three levels of hierarchy: (i) high-level activities (L1) that are independent tasks, (ii) lower-level actions (L2) that are patterns shared between activities, and (iii) fine-grained procedures (L3) that detail the exact execution steps for actions. The dataset annotations and recordings are designed so that 22.7% of L2 actions are shared between L1 activities and 14.2% of L3 procedures are shared between L2 actions. The overlap and unscripted nature of DARai allows counterfactual activities in the dataset. Experiments with various machine learning models showcase the value of DARai in uncovering important challenges in human-centered applications. Specifically, we conduct unimodal and multimodal sensor fusion experiments for recognition, temporal localization, and future action anticipation across all hierarchical annotation levels. To highlight the limitations of individual sensors, we also conduct domain-variant experiments that are enabled by DARai's multi-sensor and counterfactual activity design setup. The code, documentation, and dataset are available at the dedicated DARai website: https://alregib.ece.gatech.edu/software-and-datasets/darai-daily-activity-recordings-for-artificial-intelligence-and-machine-learning/