mlsanityData to 7 Oct 2026

RelFlexformer: Efficient Attention 3D-Transformers for Integrable Relative Positional Encodings

arXiv 2605.10706 ↗First posted 11 May 2026cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

We present a new class of efficient attention mechanisms applying universal 3D Relative Positional Encoding (RPE) methods given by arbitrary integrable modulation functions f. They lead to the new class of 3D-Transformer models, called textitRelFlexformers, flexibly integrating those RPEs, and characterized by the O(L log L) time complexity of the attention computation for the L-length input sequences. RelFlexformers builds on the theory of the Non-Uniform Fourier Transform (NU-FFT), naturally generalizing several existing efficient RPE-attention methods from structured settings with tokens homogeneously embedded in unweighted grids into general non-structured heterogeneous scenarios, where tokens' positions are arbitrarily distributed in the corresponding 3D spaces. As such, RelFlexformers can be applied in particular to model point clouds. Our extensive empirical evaluation on a large portfolio of 3D datasets confirms quality improvements provided by the NU-FFT-driven attention modulation techniques in the RelFlexformers.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 89 of its 107 references that are in the map. First posted on arXiv 11 May 2026.

Builds on

89 of 107 references.

References · 89

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,223
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2019nuScenes: A multimodal dataset for autonomous drivingcited by 4,485
  8. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  9. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
  10. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  11. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  12. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
38 more
  1. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  2. 2020Reformer: The Efficient Transformercited by 1,658
  3. 2020Rethinking Attention with Performerscited by 1,584
  4. 2020Point Transformercited by 1,462
  5. 2019KPConv: Flexible and Deformable Convolution for Point Cloudscited by 1,202
  6. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  7. 2018Self-Attention with Relative Position Representationscited by 1,112
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  9. 2018PointCNN: Convolution On X-Transformed Pointscited by 764
  10. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 747
  11. 20173D Semantic Segmentation with Submanifold Sparse Convolutional Networkscited by 746
  12. 2020PCT: Point cloud transformercited by 730
  13. 2018PointConv: Deep Convolutional Networks on 3D Point Cloudscited by 656
  14. 2021Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modelingcited by 647
  15. 2023Point Transformer V3: Simpler, Faster, Strongercited by 579
  16. 2019Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Datacited by 532
  17. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 432
  18. 2020Searching Efficient 3D Architectures with Sparse Point-Voxel Convolutioncited by 422
  19. 2020Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentationcited by 388
  20. 2017Large-scale Point Cloud Semantic Segmentation with Superpoint Graphscited by 384
  21. 2022Point Transformer V2: Grouped Vector Attention and Partition-based Poolingcited by 384
  22. 2022Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Frameworkcited by 376
  23. 2019Relation-Shape Convolutional Neural Network for Point Cloud Analysiscited by 346
  24. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  25. 2022Point-M2AE: Multi-scale Masked Autoencoders for Hierarchical Point Cloud Pre-trainingcited by 263
  26. 2018SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filterscited by 261
  27. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  28. 2020PointASNL: Robust Point Clouds Processing using Nonlocal Neural Networks with Adaptive Samplingcited by 238
  29. 2022CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformerscited by 229
  30. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  31. 2021PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Cloudscited by 211
  32. 2017SEGCloud: Semantic Segmentation of 3D Point Cloudscited by 178
  33. 2018Tangent Convolutions for Dense Prediction in 3Dcited by 173
  34. 2022Stratified Transformer for 3D Point Cloud Segmentationcited by 166
  35. 2024PointMamba: A Simple State Space Model for Point Cloud Analysiscited by 166
  36. 2021Deep Parametric Continuous Convolutional Neural Networkscited by 161
  37. 20222DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Cloudscited by 157
  38. 2022Masked Discrimination for Self-Supervised Learning on Point Cloudscited by 155

50 most cited of 89 All 89 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026ClockRoPE: Random Fourier Rotations for Temporal Routine Modelingcited by 0