# Person Segmentation and Action Classification for Multi-Channel Hemisphere Field of View LiDAR Sensors

arXiv: https://arxiv.org/abs/2411.11151 · HTML: https://mlsanity.com/paper/2411.11151/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 33 references that are in the map.
- First posted on arXiv 17 Nov 2024.

## Builds on (most cited first, 17 of 17)

- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Mask DINO: Towards A Unified Transformer-based Framework for Object Detection and Segmentation](https://mlsanity.com/paper/2206.02777/) (2022)
- [Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning](https://mlsanity.com/paper/1702.07464/) (2017)
- [Multi-step Jailbreaking Privacy Attacks on ChatGPT](https://mlsanity.com/paper/2304.05197/) (2023)
- [SalsaNext: Fast, Uncertainty-aware Semantic Segmentation of LiDAR Point Clouds for Autonomous Driving](https://mlsanity.com/paper/2003.03653/) (2020)
- [WHAM: Reconstructing World-grounded Humans with Accurate 3D Motion](https://mlsanity.com/paper/2312.07531/) (2023)
- [Segment Any Point Cloud Sequences by Distilling Vision Foundation Models](https://mlsanity.com/paper/2306.09347/) (2023)
- [DR-SPAAM: A Spatial-Attention and Auto-regressive Model for Person Detection in 2D Range Data](https://mlsanity.com/paper/2004.14079/) (2020)
- [Gimme Signals: Discriminative signal encoding for multimodal activity recognition](https://mlsanity.com/paper/2003.06156/) (2020)
- [SL-DML: Signal Level Deep Metric Learning for Multimodal One-Shot Action Recognition](https://mlsanity.com/paper/2004.11085/) (2020)
- [Onboard dynamic-object detection and tracking for autonomous robot navigation with RGB-D camera](https://mlsanity.com/paper/2303.00132/) (2023)
- [Cross-Modality Proposal-guided Feature Mining for Unregistered RGB-Thermal Pedestrian Detection](https://mlsanity.com/paper/2308.12111/) (2023)
- [Privacy-Preserving Person Detection Using Low-Resolution Infrared Cameras](https://mlsanity.com/paper/2209.11335/) (2022)

## Abstract (arXiv)

Robots need to perceive persons in their surroundings for safety and to interact with them. In this paper, we present a person segmentation and action classification approach that operates on 3D scans of hemisphere field of view LiDAR sensors. We recorded a data set with an Ouster OSDome-64 sensor consisting of scenes where persons perform three different actions and annotated it. We propose a method based on a MaskDINO model to detect and segment persons and to recognize their actions from combined spherical projected multi-channel representations of the LiDAR data with an additional positional encoding. Our approach demonstrates good performance for the person segmentation task and further performs well for the estimation of the person action states walking, waving, and sitting. An ablation study provides insights about the individual channel contributions for the person segmentation task. The trained models, code and dataset are made publicly available.