# FISHING Net: Future Inference of Semantic Heatmaps In Grids

arXiv: https://arxiv.org/abs/2006.09917 · HTML: https://mlsanity.com/paper/2006.09917/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jun 2020.

## Built on by (most cited first, 30 of 55)

- [BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers](https://mlsanity.com/paper/2203.17270/) (2022)
- [Planning-oriented Autonomous Driving](https://mlsanity.com/paper/2212.10156/) (2022)
- [FIERY: Future Instance Prediction in Bird's-Eye View from Surround Monocular Cameras](https://mlsanity.com/paper/2104.10490/) (2021)
- [NEAT: Neural Attention Fields for End-to-End Autonomous Driving](https://mlsanity.com/paper/2109.04456/) (2021)
- [Simple-BEV: What Really Matters for Multi-Sensor BEV Perception?](https://mlsanity.com/paper/2206.07959/) (2022)
- [Vision-Centric BEV Perception: A Survey](https://mlsanity.com/paper/2208.02797/) (2022)
- [BEVSegFormer: Bird's Eye View Semantic Segmentation From Arbitrary Camera Rigs](https://mlsanity.com/paper/2203.04050/) (2022)
- [Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipe](https://mlsanity.com/paper/2209.05324/) (2022)
- [Structured Bird's-Eye-View Traffic Scene Understanding from Onboard Images](https://mlsanity.com/paper/2110.01997/) (2021)
- [KING: Generating Safety-Critical Driving Scenarios for Robust Imitation via Kinematics Gradients](https://mlsanity.com/paper/2204.13683/) (2022)
- [Cam4DOcc: Benchmark for Camera-Only 4D Occupancy Forecasting in Autonomous Driving Applications](https://mlsanity.com/paper/2311.17663/) (2023)
- [StretchBEV: Stretching Future Instance Prediction Spatially and Temporally](https://mlsanity.com/paper/2203.13641/) (2022)
- [Topology Preserving Local Road Network Estimation from Single Onboard Camera Image](https://mlsanity.com/paper/2112.10155/) (2021)
- [LaRa: Latents and Rays for Multi-Camera Bird's-Eye-View Semantic Segmentation](https://mlsanity.com/paper/2206.13294/) (2022)
- [BEVCar: Camera-Radar Fusion for BEV Map and Object Segmentation](https://mlsanity.com/paper/2403.11761/) (2024)
- [MapDistill: Boosting Efficient Camera-based HD Map Construction via Camera-LiDAR Fusion Model Distillation](https://mlsanity.com/paper/2407.11682/) (2024)
- [Grid-Centric Traffic Scenario Perception for Autonomous Driving: A Comprehensive Review](https://mlsanity.com/paper/2303.01212/) (2023)
- [Learning by Watching](https://mlsanity.com/paper/2106.05966/) (2021)
- [MarS3D: A Plug-and-Play Motion-Aware Model for Semantic Segmentation on Multi-Scan 3D Point Clouds](https://mlsanity.com/paper/2307.09316/) (2023)
- [LiDAR2Map: In Defense of LiDAR-Based Semantic Map Construction Using Online Camera Distillation](https://mlsanity.com/paper/2304.11379/) (2023)
- [CoBEV: Elevating Roadside 3D Object Detection with Depth and Height Complementarity](https://mlsanity.com/paper/2310.02815/) (2023)
- [OccFeat: Self-supervised Occupancy Feature Prediction for Pretraining BEV Segmentation Networks](https://mlsanity.com/paper/2404.14027/) (2024)
- [DeepFusion: A Robust and Modular 3D Object Detector for Lidars, Cameras and Radars](https://mlsanity.com/paper/2209.12729/) (2022)
- [NVRadarNet: Real-Time Radar Obstacle and Free Space Detection for Autonomous Driving](https://mlsanity.com/paper/2209.14499/) (2022)
- [Reimagining an autonomous vehicle](https://mlsanity.com/paper/2108.05805/) (2021)
- [NeMO: Neural Map Growing System for Spatiotemporal Fusion in Bird's-Eye-View and BDD-Map Benchmark](https://mlsanity.com/paper/2306.04540/) (2023)
- [A Technical Survey and Evaluation of Traditional Point Cloud Clustering Methods for LiDAR Panoptic Segmentation](https://mlsanity.com/paper/2108.09522/) (2021)
- [DualCross: Cross-Modality Cross-Domain Adaptation for Monocular BEV Perception](https://mlsanity.com/paper/2305.03724/) (2023)
- [Improving Online Lane Graph Extraction by Object-Lane Clustering](https://mlsanity.com/paper/2307.10947/) (2023)
- [Real-time 3D semantic occupancy prediction for autonomous vehicles using memory-efficient sparse convolution](https://mlsanity.com/paper/2403.08748/) (2024)

## Abstract (arXiv)

For autonomous robots to navigate a complex environment, it is crucial to understand the surrounding scene both geometrically and semantically. Modern autonomous robots employ multiple sets of sensors, including lidars, radars, and cameras. Managing the different reference frames and characteristics of the sensors, and merging their observations into a single representation complicates perception. Choosing a single unified representation for all sensors simplifies the task of perception and fusion. In this work, we present an end-to-end pipeline that performs semantic segmentation and short term prediction using a top-down representation. Our approach consists of an ensemble of neural networks which take in sensor data from different sensor modalities and transform them into a single common top-down semantic grid representation. We find this representation favorable as it is agnostic to sensor-specific reference frames and captures both the semantic and geometric information for the surrounding scene. Because the modalities share a single output representation, they can be easily aggregated to produce a fused output. In this work we predict short-term semantic grids but the framework can be extended to other tasks. This approach offers a simple, extensible, end-to-end approach for multi-modal perception and prediction.