# RELLIS-3D Dataset: Data, Benchmarks and Analysis

arXiv: https://arxiv.org/abs/2011.12954 · HTML: https://mlsanity.com/paper/2011.12954/

## Key facts

- Cited by 119 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Nov 2020.

## Built on by (most cited first, 30 of 119)

- [Segment Any Point Cloud Sequences by Distilling Vision Foundation Models](https://mlsanity.com/paper/2306.09347/) (2023)
- [Multi-modal Sensor Fusion for Auto Driving Perception: A Survey](https://mlsanity.com/paper/2202.02703/) (2022)
- [Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Driving](https://mlsanity.com/paper/2405.05258/) (2024)
- [GANav: Efficient Terrain Segmentation for Robot Navigation in Unstructured Outdoor Environments](https://mlsanity.com/paper/2103.04233/) (2021)
- [The GOOSE Dataset for Perception in Unstructured Environments](https://mlsanity.com/paper/2310.16788/) (2023)
- [TartanGround: A Large-Scale Dataset for Ground Robot Perception and Navigation](https://mlsanity.com/paper/2505.10696/) (2025)
- [ORFD: A Dataset and Benchmark for Off-Road Freespace Detection](https://mlsanity.com/paper/2206.09907/) (2022)
- [4D Contrastive Superflows are Dense 3D Representation Learners](https://mlsanity.com/paper/2407.06190/) (2024)
- [Towards Robust Robot 3D Perception in Urban Environments: The UT Campus Object Dataset](https://mlsanity.com/paper/2309.13549/) (2023)
- [The Oxford Spires Dataset: Benchmarking Large-Scale LiDAR-Visual Localisation, Reconstruction and Radiance Field Methods](https://mlsanity.com/paper/2411.10546/) (2024)
- [Learning Off-Road Terrain Traversability with Self-Supervisions Only](https://mlsanity.com/paper/2305.18896/) (2023)
- [UAVScenes: A Multi-Modal Dataset for UAVs](https://mlsanity.com/paper/2507.22412/) (2025)
- [RoadRunner -- Learning Traversability Estimation for Autonomous Off-road Driving](https://mlsanity.com/paper/2402.19341/) (2024)
- [Is Your LiDAR Placement Optimized for 3D Scene Understanding?](https://mlsanity.com/paper/2403.17009/) (2024)
- [Multi-Space Alignments Towards Universal LiDAR Segmentation](https://mlsanity.com/paper/2405.01538/) (2024)
- [What a MESS: Multi-Domain Evaluation of Zero-Shot Semantic Segmentation](https://mlsanity.com/paper/2306.15521/) (2023)
- [Self-Supervised Traversability Prediction by Learning to Reconstruct Safe Terrain](https://mlsanity.com/paper/2208.01329/) (2022)
- [ScaTE: A Scalable Framework for Self-Supervised Traversability Estimation in Unstructured Environments](https://mlsanity.com/paper/2209.06522/) (2022)
- [OFFSEG: A Semantic Segmentation Framework For Off-Road Driving](https://mlsanity.com/paper/2103.12417/) (2021)
- [Risk-Aware Off-Road Navigation via a Learned Speed Distribution Map](https://mlsanity.com/paper/2203.13429/) (2022)
- [Deep Learning for Camera Calibration and Beyond: A Survey](https://mlsanity.com/paper/2303.10559/) (2023)
- [LiMoE: Mixture of LiDAR Representation Learners from Automotive Scenes](https://mlsanity.com/paper/2501.04004/) (2025)
- [WildScenes: A Benchmark for 2D and 3D Semantic Segmentation in Large-scale Natural Environments](https://mlsanity.com/paper/2312.15364/) (2023)
- [Learning Risk-Aware Costmaps via Inverse Reinforcement Learning for Off-Road Navigation](https://mlsanity.com/paper/2302.00134/) (2023)
- [CalibDNN: Multimodal Sensor Calibration for Perception Using Deep Neural Networks](https://mlsanity.com/paper/2103.14793/) (2021)
- [Online Segmentation of LiDAR Sequences: Dataset and Algorithm](https://mlsanity.com/paper/2206.08194/) (2022)
- [GaussianFormer3D: Multi-Modal Gaussian-based Semantic Occupancy Prediction with 3D Deformable Attention](https://mlsanity.com/paper/2505.10685/) (2025)
- [Excavating in the Wild: The GOOSE-Ex Dataset for Semantic Segmentation](https://mlsanity.com/paper/2409.18788/) (2024)
- [LargeAD: Large-Scale Cross-Sensor Data Pretraining for Autonomous Driving](https://mlsanity.com/paper/2501.04005/) (2025)
- [3EED: Ground Everything Everywhere in 3D](https://mlsanity.com/paper/2511.01755/) (2025)

## Abstract (arXiv)

Semantic scene understanding is crucial for robust and safe autonomous navigation, particularly so in off-road environments. Recent deep learning advances for 3D semantic segmentation rely heavily on large sets of training data, however existing autonomy datasets either represent urban environments or lack multimodal off-road data. We fill this gap with RELLIS-3D, a multimodal dataset collected in an off-road environment, which contains annotations for 13,556 LiDAR scans and 6,235 images. The data was collected on the Rellis Campus of Texas A&M University and presents challenges to existing algorithms related to class imbalance and environmental topography. Additionally, we evaluate the current state-of-the-art deep learning semantic segmentation models on this dataset. Experimental results show that RELLIS-3D presents challenges for algorithms designed for segmentation in urban environments. This novel dataset provides the resources needed by researchers to continue to develop more advanced algorithms and investigate new research directions to enhance autonomous navigation in off-road environments. RELLIS-3D is available at https://github.com/unmannedlab/RELLIS-3D