mlsanityData to 14 Oct 2026

BiSHop: Bi-Directional Cellular Learning for Tabular Data with Generalized Sparse Modern Hopfield Model

arXiv 2404.03830 ↗First posted 4 Apr 2024cs.LG cs.AI stat.MLcited by 21

Not on the map. Why?

Abstract (arXiv)

We introduce the Bi-Directional Sparse Hopfield Network (BiSHop), a novel end-to-end framework for deep tabular learning. BiSHop handles the two major challenges of deep tabular learning: non-rotationally invariant data structure and feature sparsity in tabular data. Our key motivation comes from the recent established connection between associative memory and attention mechanisms. Consequently, BiSHop uses a dual-component approach, sequentially processing data both column-wise and row-wise through two interconnected directional learning modules. Computationally, these modules house layers of generalized sparse modern Hopfield layers, a sparse extension of the modern Hopfield model with adaptable sparsity. Methodologically, BiSHop facilitates multi-scale representation learning, capturing both intra-feature and inter-feature interactions, with adaptive sparsity at each scale. Empirically, through experiments on diverse real-world datasets, we demonstrate that BiSHop surpasses current SOTA methods with significantly less HPO runs, marking it a robust solution for deep tabular learning.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 54 of its 89 references that are in the map. First posted on arXiv 4 Apr 2024.

Builds on

54 of 89 references.

References · 54

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2020Longformer: The Long-Document Transformercited by 3,413
  7. 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,169
  8. 2022A Time Series is Worth 64 Words: Long-term Forecasting with Transformerscited by 2,039
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  10. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  11. 2020Efficient Transformers: A Surveycited by 790
  12. 2021Revisiting Deep Learning Models for Tabular Datacited by 740
38 more
  1. 2022TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Secondcited by 674
  2. 2019TabNet: Attentive Interpretable Tabular Learningcited by 648
  3. 2021Deep Neural Networks and Tabular Data: A Surveycited by 491
  4. 2020Hopfield Networks is All You Needcited by 475
  5. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 409
  6. 2021SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Trainingcited by 319
  7. 2016Dense Associative Memory for Pattern Recognitioncited by 236
  8. 2023When Do Neural Nets Outperform Boosted Trees on Tabular Data?cited by 223
  9. 2019Neural Oblivious Decision Ensembles for Deep Learning on Tabular Datacited by 203
  10. 2022On Embeddings for Numerical Features in Tabular Deep Learningcited by 186
  11. 2019Sparse Sequence-to-Sequence Modelscited by 170
  12. 2023Birth of a Transformer: A Memory Viewpointcited by 161
  13. 2019Blockwise Self-Attention for Long Document Understandingcited by 159
  14. 2019Adaptively Sparse Transformerscited by 155
  15. 2021Well-tuned Simple Nets Excel on Tabular Datasetscited by 151
  16. 2022FiLM: Frequency improved Legendre Memory Model for Long-term Time Series Forecastingcited by 147
  17. 2023DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genomecited by 143
  18. 2020Large Associative Memory Problem in Neurobiology and Machine Learningcited by 112
  19. 2021CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIPcited by 89
  20. 2020Tabular Transformers for Modeling Multivariate Time Seriescited by 77
  21. 2023Energy Transformercited by 71
  22. 2023Benchmarking Distribution Shift in Tabular Data with TableShiftcited by 66
  23. 2020Modern Hopfield Networks and Attention for Immune Repertoire Classificationcited by 53
  24. 2022A Theoretical Analysis on Feature Learning in Neural Networks: Emergence from Inputs and Advantage over Fixed Featurescited by 49
  25. 2023Conformal Prediction for Time Series with Modern Hopfield Networkscited by 42
  26. 2023On Sparse Modern Hopfield Modelcited by 42
  27. 2022T2G-Former: Organizing Tabular Features into Relation Graphs Promotes Heterogeneous Feature Interactioncited by 41
  28. 2024On Computational Limits of Modern Hopfield Models: A Fine-Grained Complexity Analysiscited by 41
  29. 2022History Compression via Language Models in Reinforcement Learningcited by 39
  30. 2023Scaling Laws for Associative Memoriescited by 35
  31. 2023STanHop: Sparse Tandem Hopfield Model for Memory-Enhanced Time Series Predictioncited by 35
  32. 2021Hierarchical Associative Memorycited by 34
  33. 2023TANGOS: Regularizing Tabular Neural Networks through Gradient Orthogonalization and Specializationcited by 32
  34. 2022PTab: Using the Pre-trained Language Model for Modeling Tabular Datacited by 30
  35. 2023The Trade-off between Universality and Label Efficiency of Representations from Contrastive Learningcited by 27
  36. 2020DNF-Net: A Neural Architecture for Tabular Datacited by 17
  37. 2023Simplicial Hopfield networkscited by 16
  38. 2023Long Sequence Hopfield Memorycited by 16

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 15; 2025: 5; 2026: 1.

21 papers.

Sort:

By area:

  1. 2024TALENT: A Tabular Analytics and Learning Toolboxcited by 52
  2. 2025Representation Learning for Tabular Data: A Comprehensive Surveycited by 36
  3. 2024Computational Limits of Low-Rank Adaptation (LoRA) Fine-Tuning for Transformer Modelscited by 31
  4. 2024PTaRL: Prototype-based Tabular Representation Learning via Space Calibrationcited by 28
  5. 2024Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Timecited by 22
  6. 2024Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descentcited by 22
  7. 2024The Computational Limits of State-Space Models and Mamba via the Lens of Circuit Complexitycited by 21
  8. 2024Conv-CoA: Improving Open-domain Question Answering in Large Language Models via Conversational Chain-of-Actioncited by 19
  9. 2024HSR-Enhanced Sparse Attention Accelerationcited by 19
  10. 2024Provably Optimal Memory Capacity for Modern Hopfield Models: Transformer-Compatible Dense Associative Memories as Spherical Codescited by 19
  11. 2024A Survey on Deep Tabular Learningcited by 18
  12. 2024Nonparametric Modern Hopfield Modelscited by 16
9 more
  1. 2024Provable Differentially Private Computation of the Cross-Attention Mechanismcited by 14
  2. 2024Theoretical Constraints on the Expressive Power of mathsfRoPE-based Tensor Attention Transformerscited by 12
  3. 2024Modern Hopfield Networks Require Chain-of-Thought to Solve mathsfNC1-Hard Problemscited by 9
  4. 2025Fast and Low-Cost Genomic Foundation Models via Outlier Removalcited by 8
  5. 2024Unlocking the Theory Behind Scaling 1-Bit Neural Networkscited by 4
  6. 2025Neural Algorithmic Reasoning for Hypergraphs with Looped Transformerscited by 4
  7. 2025Time and Memory Trade-off of KV-Cache Compression in Tensor Transformer Decodingcited by 2
  8. 2025Multi-branch of Attention Yields Accurate Results for Tabular Datacited by 0
  9. 2026TabNSM: Neural Sparse Mixer for Tabular Regressioncited by 0