# FT-HID: A Large Scale RGB-D Dataset for First and Third Person Human Interaction Analysis

arXiv: https://arxiv.org/abs/2209.10155 · HTML: https://mlsanity.com/paper/2209.10155/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 94 references that are in the map.
- First posted on arXiv 21 Sep 2022.

## Builds on (most cited first, 22 of 22)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations](https://mlsanity.com/paper/1704.02463/) (2017)
- [Actional-Structural Graph Convolutional Networks for Skeleton-based Action Recognition](https://mlsanity.com/paper/1904.12659/) (2019)
- [Cross-view Action Modeling, Learning and Recognition](https://mlsanity.com/paper/1405.2941/) (2014)
- [View Adaptive Neural Networks for High Performance Skeleton-based Human Action Recognition](https://mlsanity.com/paper/1804.07453/) (2018)
- [Modality Distillation with Multiple Stream Networks for Action Recognition](https://mlsanity.com/paper/1806.07110/) (2018)
- [Rank Pooling for Action Recognition](https://mlsanity.com/paper/1512.01848/) (2015)
- [Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos](https://mlsanity.com/paper/1603.07120/) (2016)
- [HOPC: Histogram of Oriented Principal Components of 3D Pointclouds for Action Recognition](https://mlsanity.com/paper/1408.3809/) (2014)
- [Investigation of Different Skeleton Features for CNN-based 3D Action Recognition](https://mlsanity.com/paper/1705.00835/) (2017)
- [Cooperative Training of Deep Aggregation Networks for RGB-D Action Recognition](https://mlsanity.com/paper/1801.01080/) (2017)
- [Action Recognition Based on Joint Trajectory Maps with Convolutional Neural Networks](https://mlsanity.com/paper/1612.09401/) (2016)
- [Generalized Rank Pooling for Activity Recognition](https://mlsanity.com/paper/1704.02112/) (2017)
- [A Two-stream Neural Network for Pose-based Hand Gesture Recognition](https://mlsanity.com/paper/2101.08926/) (2021)
- [Egocentric Activity Recognition with Multimodal Fisher Vector](https://mlsanity.com/paper/1601.06603/) (2016)
- [Viewpoint Invariant Action Recognition using RGB-D Videos](https://mlsanity.com/paper/1709.05087/) (2017)
- [3D Human Activity Recognition with Reconfigurable Convolutional Neural Networks](https://mlsanity.com/paper/1501.06262/) (2015)

## Built on by (most cited first, 3 of 3)

- [PlayerOne: Egocentric World Simulator](https://mlsanity.com/paper/2506.09995/) (2025)
- [EgoSim: Egocentric World Simulator for Embodied Interaction Generation](https://mlsanity.com/paper/2604.01001/) (2026)
- [Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Vision](https://mlsanity.com/paper/2506.06253/) (2025)

## Abstract (arXiv)

Analysis of human interaction is one important research topic of human motion analysis. It has been studied either using first person vision (FPV) or third person vision (TPV). However, the joint learning of both types of vision has so far attracted little attention. One of the reasons is the lack of suitable datasets that cover both FPV and TPV. In addition, existing benchmark datasets of either FPV or TPV have several limitations, including the limited number of samples, participant subjects, interaction categories, and modalities. In this work, we contribute a large-scale human interaction dataset, namely, FT-HID dataset. FT-HID contains pair-aligned samples of first person and third person visions. The dataset was collected from 109 distinct subjects and has more than 90K samples for three modalities. The dataset has been validated by using several existing action recognition methods. In addition, we introduce a novel multi-view interaction mechanism for skeleton sequences, and a joint learning multi-stream framework for first person and third person visions. Both methods yield promising results on the FT-HID dataset. It is expected that the introduction of this vision-aligned large-scale dataset will promote the development of both FPV and TPV, and their joint learning techniques for human action analysis. The dataset and code are available at hrefhttps://github.com/ENDLICHERE/FT-HIDhere.