mlsanityData to 7 Oct 2026

Locality-Sensitive Hashing-Based Efficient Point Transformer with Applications in High-Energy Physics

arXiv 2402.12535 ↗First posted 19 Feb 2024cs.LG hep-excited by 12

Not on the map. Why?

Abstract (arXiv)

This study introduces a novel transformer model optimized for large-scale point cloud processing in scientific domains such as high-energy physics (HEP) and astrophysics. Addressing the limitations of graph neural networks and standard transformers, our model integrates local inductive bias and achieves near-linear complexity with hardware-friendly regular operations. One contribution of this work is the quantitative analysis of the error-complexity tradeoff of various sparsification techniques for building efficient transformers. Our findings highlight the superiority of using locality-sensitive hashing (LSH), especially OR & AND-construction LSH, in kernel approximation for large-scale point cloud data with local inductive bias. Based on this finding, we propose LSH-based Efficient Point Transformer (HEPT), which combines E2LSH with OR & AND constructions and is built upon regular computations. HEPT demonstrates remarkable performance on two critical yet time-consuming HEP tasks, significantly outperforming existing GNNs and transformers in accuracy and computational speed, marking a significant advancement in geometric deep learning and large-scale scientific data processing. Our code is available at https://github.com/Graph-COM/HEPT.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 59 of its 91 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

59 of 91 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2016Layer Normalizationcited by 5,579
  6. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  9. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,414
  10. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  12. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
38 more
  1. 2020Reformer: The Efficient Transformercited by 1,658
  2. 2020Rethinking Attention with Performerscited by 1,584
  3. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  4. 2020Point Transformercited by 1,462
  5. 2021Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gaugescited by 1,131
  6. 2018Self-Attention with Relative Position Representationscited by 1,112
  7. 2016Geometric deep learning: going beyond Euclidean datacited by 1,042
  8. 2021E(n) Equivariant Graph Neural Networkscited by 951
  9. 2015Gated Graph Sequence Neural Networkscited by 792
  10. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  11. 2023Point Transformer V3: Simpler, Faster, Strongercited by 579
  12. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
  13. 2021Random Feature Attentioncited by 350
  14. 2020Learning from Protein Structure with Geometric Vector Perceptronscited by 308
  15. 2020Sparse Sinkhorn Attentioncited by 250
  16. 2021Voxel Transformer for 3D Object Detectioncited by 247
  17. 2023NodeFormer: A Scalable Graph Structure Learning Transformer for Node Classificationcited by 207
  18. 2021Rethinking and Improving Relative Position Encoding for Vision Transformercited by 206
  19. 2021Embracing Single Stride 3D Object Detector with Sparse Transformercited by 193
  20. 2023Exphormer: Sparse Transformers for Graphscited by 166
  21. 2022EquiBind: Geometric Deep Learning for Drug Binding Structure Predictioncited by 143
  22. 2022NAGphormer: A Tokenized Graph Transformer for Node Classification in Large Graphscited by 137
  23. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 128
  24. 2023DSVT: Dynamic Sparse Voxel Transformer with Rotated Setscited by 122
  25. 2014Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS)cited by 109
  26. 2015Practical and Optimal LSH for Angular Distancecited by 105
  27. 2022Particle Transformer for Jet Taggingcited by 95
  28. 2022SWFormer: Sparse Window Transformer for 3D Object Detection in Point Cloudscited by 92
  29. 2023HyperAttention: Long-context Attention in Near-Linear Timecited by 85
  30. 2020Computing Graph Neural Networks: A Survey from Algorithms to Acceleratorscited by 81
  31. 2015On the Error of Random Fourier Featurescited by 76
  32. 2022Hierarchical Graph Transformer with Adaptive Node Samplingcited by 72
  33. 2023FlatFormer: Flattened Window Attention for Efficient Point Cloud Transformercited by 72
  34. 2021Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encodingcited by 50
  35. 2021Particle Cloud Generation with Message Passing Generative Adversarial Networkscited by 45
  36. 2019Learning representations of irregular particle-detector geometry with distance-weighted graph networkscited by 43
  37. 2020SMYRF: Efficient Attention using Asymmetric Clusteringcited by 43
  38. 2022Evaluating generative models in high energy physicscited by 31

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 8; 2026: 2.

12 papers.

  1. 2024What Can We Learn from State Space Models for Machine Learning on Graphs?cited by 13
  2. 2025Particle Trajectory Representation Learning with Masked Point Modelingcited by 8
  3. 2025hls4ml: A Flexible, Open-Source Platform for Deep Learning Acceleration on Reconfigurable Hardwarecited by 8
  4. 2024Lorentz-Equivariant Quantum Graph Neural Network for High-Energy Physicscited by 4
  5. 2025Spatially Aware Linear Transformer (SAL-T) for Particle Jet Taggingcited by 3
  6. 2025Structural Alignment Improves Graph Test-Time Adaptationcited by 2
  7. 2025InterMamba: Efficient Human-Human Interaction Generation with Adaptive Spatio-Temporal Mambacited by 2
  8. 2025BSA: Ball Sparse Attention for Large-scale Geometriescited by 1
  9. 2025Locality-Sensitive Hashing-Based Efficient Point Transformer for Charged Particle Reconstructioncited by 1
  10. 2025From Small to Large: Generalization Bounds for Transformers on Variable-Size Inputscited by 1
  11. 2026Patch Hierarchical Attention Transformer for Efficient Particle Jet Taggingcited by 0
  12. 2026HEPTv2: End-to-End Efficient Point Transformer for Charged Particle Reconstructioncited by 0