OAFuser: Towards Omni-Aperture Fusion for Light Field Semantic Segmentation
Not on the map. Why?
Abstract (arXiv)
Light field cameras are capable of capturing intricate angular and spatial details. This allows for acquiring complex light patterns and details from multiple angles, significantly enhancing the precision of image semantic segmentation. However, two significant issues arise: (1) The extensive angular information of light field cameras contains a large amount of redundant data, which is overwhelming for the limited hardware resources of intelligent agents. (2) A relative displacement difference exists in the data collected by different micro-lenses. To address these issues, we propose an Omni-Aperture Fusion model (OAFuser) that leverages dense context from the central view and extracts the angular information from sub-aperture images to generate semantically consistent results. To simultaneously streamline the redundant information from the light field cameras and avoid feature loss during network propagation, we present a simple yet very effective Sub-Aperture Fusion Module (SAFM). This module efficiently embeds sub-aperture images in angular features, allowing the network to process each sub-aperture image with a minimal computational demand of only (around 1GFlops). Furthermore, to address the mismatched spatial information across viewpoints, we present a Center Angular Rectification Module (CARM) to realize feature resorting and prevent feature occlusion caused by misalignment. The proposed OAFuser achieves state-of-the-art performance on four UrbanLF datasets in terms of all evaluation metrics and sets a new record of 84.93% in mIoU on the UrbanLF-Real Extended dataset, with a gain of +3.69%. The source code for OAFuser is available at https://github.com/FeiBryantkit/OAFuser.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 42 of its 77 references that are in the map. First posted on arXiv 28 Jul 2023.
Builds on
42 of 77 references.
References · 42
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
- 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
- 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
- 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,223
- 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
- 2016Pyramid Scene Parsing Networkcited by 2,479
- 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,487
- 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
- 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
- 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
- 2018Dual Attention Network for Scene Segmentationcited by 1,003
30 more
- 2019Segmentation Transformer: Object-Contextual Representations for Semantic Segmentationcited by 576
- 2016RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentationcited by 465
- 2022SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentationcited by 252
- 2021Exploring Cross-Image Pixel Contrast for Semantic Segmentationcited by 242
- 2022CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformerscited by 231
- 2020MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learningcited by 157
- 2020Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentationcited by 122
- 2023Delivering Arbitrary-Modal Semantic Segmentationcited by 117
- 2019ACNet: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentationcited by 107
- 2020Efficient RGB-D Semantic Segmentation for Indoor Scene Analysiscited by 84
- 2018Accel: A Corrective Fusion Network for Efficient Semantic Segmentation on Videocited by 78
- 2020Temporally Distributed Networks for Fast Video Semantic Segmentationcited by 77
- 2021Temporal Memory Attention for Video Semantic Segmentationcited by 50
- 2022SuperYOLO: Super Resolution Assisted Object Detection in Multimodal Remote Sensing Imagerycited by 47
- 2022Learning to Deblur using Light Field Generated and Real Defocus Imagescited by 46
- 2022VSA: Learning Varied-Size Window Attention in Vision Transformerscited by 43
- 2022Disentangling Light Fields for Super-Resolution and Disparity Estimationcited by 42
- 2019Spatial-Angular Interaction for Light Field Image Super-Resolutioncited by 38
- 2020Light Field Spatial Super-resolution via Deep Combinatorial Geometry Embedding and Structural Consistency Regularizationcited by 36
- 2022NestedFormer: Nested Modality-Aware Transformer for Brain Tumor Segmentationcited by 34
- 2022HRFuser: A Multi-resolution Sensor Fusion Architecture for 2D Object Detectioncited by 31
- 2021Exploring Event-driven Dynamic Context for Accident Scene Segmentationcited by 23
- 2021Trans4Trans: Efficient Transformer for Transparent Object and Semantic Scene Segmentation in Real-World Navigation Assistancecited by 22
- 2023Learning Non-Local Spatial-Angular Correlation for Light Field Image Super-Resolutioncited by 21
- 2020Video Semantic Segmentation with Distortion-Aware Feature Correctioncited by 9
- 2020Spatial-Angular Attention Network for Light Field Reconstructioncited by 9
- 2018MMFNet: A Multi-modality MRI Fusion Network for Segmentation of Nasopharyngeal Carcinomacited by 8
- 2022Learning from Pixel-Level Noisy Label : A New Perspective for Light Field Saliency Detectioncited by 5
- 2022Surrogate-assisted Multi-objective Neural Architecture Search for Real-time Semantic Segmentationcited by 5
- 2023Test-Time Adaptation for Nighttime Color-Thermal Semantic Segmentationcited by 4
Built on by
Citing papers per year: 2025: 1.
1 paper.