# Toyota Smarthome Untrimmed: Real-World Untrimmed Videos for Activity Detection

arXiv: https://arxiv.org/abs/2010.14982 · HTML: https://mlsanity.com/paper/2010.14982/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Oct 2020.

## Built on by (most cited first, 30 of 42)

- [MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detection](https://mlsanity.com/paper/2112.03902/) (2021)
- [Spotting Temporally Precise, Fine-Grained Events in Video](https://mlsanity.com/paper/2207.10213/) (2022)
- [Learning an Augmented RGB Representation with Cross-Modal Knowledge Distillation for Action Detection](https://mlsanity.com/paper/2108.03619/) (2021)
- [Dual DETRs for Multi-Label Temporal Action Detection](https://mlsanity.com/paper/2404.00653/) (2024)
- [Let's Play for Action: Recognizing Activities of Daily Living by Learning from Life Simulation Video Games](https://mlsanity.com/paper/2107.05617/) (2021)
- [Navigating Open Set Scenarios for Skeleton-based Action Recognition](https://mlsanity.com/paper/2312.06330/) (2023)
- [CTRN: Class-Temporal Relational Network for Action Detection](https://mlsanity.com/paper/2110.13473/) (2021)
- [LAC: Latent Action Composition for Skeleton-based Action Segmentation](https://mlsanity.com/paper/2308.14500/) (2023)
- [ATTACH Dataset: Annotated Two-Handed Assembly Actions for Human Action Understanding](https://mlsanity.com/paper/2304.08210/) (2023)
- [Computer Vision for Primate Behavior Analysis in the Wild](https://mlsanity.com/paper/2401.16424/) (2024)
- [LLAVIDAL: A Large LAnguage VIsion Model for Daily Activities of Living](https://mlsanity.com/paper/2406.09390/) (2024)
- [Robust Action Segmentation from Timestamp Supervision](https://mlsanity.com/paper/2210.06501/) (2022)
- [Self-Supervised Skeleton-Based Action Representation Learning: A Benchmark and Beyond](https://mlsanity.com/paper/2406.02978/) (2024)
- [Kaiwu: A Multimodal Manipulation Dataset and Framework for Robot Learning and Human-Robot Interaction](https://mlsanity.com/paper/2503.05231/) (2025)
- [Are Visual-Language Models Effective in Action Recognition? A Comparative Study](https://mlsanity.com/paper/2410.17149/) (2024)
- [Enhancing Action Recognition by Leveraging the Hierarchical Structure of Actions and Textual Context](https://mlsanity.com/paper/2410.21275/) (2024)
- [From My View to Yours: Learning Egocentric Cues from Exocentric Video using Privileged Egocentric Supervision](https://mlsanity.com/paper/2501.05711/) (2025)
- [Action Anticipation at a Glimpse: To What Extent Can Multimodal Cues Replace Video?](https://mlsanity.com/paper/2512.02846/) (2025)
- [Human-Centric Open-Future Task Discovery: Formulation, Benchmark, and Scalable Tree-Based Search](https://mlsanity.com/paper/2511.18929/) (2025)
- [Multimodal Generation of Novel Action Appearances for Synthetic-to-Real Recognition of Activities of Daily Living](https://mlsanity.com/paper/2208.01910/) (2022)
- [MS-Temba: Multi-Scale Temporal Mamba for Understanding Long Untrimmed Videos](https://mlsanity.com/paper/2501.06138/) (2025)
- [Pose-Aware Weakly-Supervised Action Segmentation](https://mlsanity.com/paper/2504.05700/) (2025)
- [Synthetic Human Action Video Data Generation with Pose Transfer](https://mlsanity.com/paper/2506.09411/) (2025)
- [DuoCLR: Dual-Surrogate Contrastive Learning for Skeleton-based Human Action Segmentation](https://mlsanity.com/paper/2509.05543/) (2025)
- [VisCoP: Visual Probing for Video Domain Adaptation of Vision Language Models](https://mlsanity.com/paper/2510.13808/) (2025)
- [Understanding Multimodal Complementarity for Single-Frame Action Anticipation](https://mlsanity.com/paper/2601.22039/) (2026)
- [MM-SEAL: A Large-scale Video Dataset of Multi-person Multi-grained Spatio-temporally Action Localization](https://mlsanity.com/paper/2204.02688/) (2022)
- [Co-Occurrence Matters: Learning Action Relation for Temporal Action Localization](https://mlsanity.com/paper/2303.08463/) (2023)
- [AAN: Attributes-Aware Network for Temporal Action Detection](https://mlsanity.com/paper/2309.00696/) (2023)
- [P-Age: Pexels Dataset for Robust Spatio-Temporal Apparent Age Classification](https://mlsanity.com/paper/2311.02432/) (2023)

## Abstract (arXiv)

Designing activity detection systems that can be successfully deployed in daily-living environments requires datasets that pose the challenges typical of real-world scenarios. In this paper, we introduce a new untrimmed daily-living dataset that features several real-world challenges: Toyota Smarthome Untrimmed (TSU). TSU contains a wide variety of activities performed in a spontaneous manner. The dataset contains dense annotations including elementary, composite activities and activities involving interactions with objects. We provide an analysis of the real-world challenges featured by our dataset, highlighting the open issues for detection algorithms. We show that current state-of-the-art methods fail to achieve satisfactory performance on the TSU dataset. Therefore, we propose a new baseline method for activity detection to tackle the novel challenges provided by our dataset. This method leverages one modality (i.e. optic flow) to generate the attention weights to guide another modality (i.e RGB) to better detect the activity boundaries. This is particularly beneficial to detect activities characterized by high temporal variance. We show that the method we propose outperforms state-of-the-art methods on TSU and on another popular challenging dataset, Charades.