mlsanityData to 14 Oct 2026

Learning long-range spatial dependencies with horizontal gated-recurrent units

arXiv 1805.08315 ↗First posted 21 May 2018cs.CVcited by 98

Not on the map. Why?

Abstract (arXiv)

Progress in deep learning has spawned great successes in many engineering applications. As a prime example, convolutional neural networks, a type of feedforward neural networks, are now approaching -- and sometimes even surpassing -- human accuracy on a variety of visual recognition tasks. Here, however, we show that these neural networks and their recent extensions struggle in recognition tasks where co-dependent visual features must be detected over long spatial ranges. We introduce the horizontal gated-recurrent unit (hGRU) to learn intrinsic horizontal connections -- both within and across feature columns. We demonstrate that a single hGRU layer matches or outperforms all tested feedforward hierarchical baselines including state-of-the-art architectures which have orders of magnitude more free parameters. We further discuss the biological plausibility of the hGRU in comparison to anatomical data from the visual cortex as well as human behavioral data on a classic contour detection task.

Key facts. Cited by 98 papers in the map (data through 14 Oct 2026). First posted on arXiv 21 May 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 19; 2022: 18; 2023: 21; 2024: 22; 2025: 11; 2026: 7.

98 papers.

Sort:

By area:

  1. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  2. 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 471
  3. 2022cosFormer: Rethinking Softmax in Attentioncited by 213
  4. 2022Mega: Moving Average Equipped Gated Attentioncited by 179
  5. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 130
  6. 2023Circuit Component Reuse Across Tasks in Transformer Language Modelscited by 108
  7. 2021Luna: Linear Unified Nested Attentioncited by 96
  8. 2022What Makes Convolutional Models Great on Long Sequence Modeling?cited by 87
  9. 2022Flowformer: Linearizing Transformers with Conservation Flowscited by 69
  10. 2021Skyformer: Remodel Self-Attention with Gaussian Kernel and Nystr"om Methodcited by 63
  11. 2024Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Lengthcited by 46
  12. 2022A Benchmark for Compositional Visual Reasoningcited by 43
38 more
  1. 2023Neither hype nor gloom do DNNs justicecited by 40
  2. 2022Efficient Long Sequence Modeling via State Space Augmented Transformercited by 37
  3. 2023Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priorscited by 34
  4. 2025PolaFormer: Polarity-aware Linear Attention for Vision Transformerscited by 32
  5. 2022Linear Complexity Randomized Self-attention Mechanismcited by 31
  6. 2023Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representationcited by 29
  7. 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
  8. 2021FMMformer: Efficient and Flexible Transformer via Decomposed Near-field and Far-field Attentioncited by 26
  9. 2022Simplifying and Understanding State Space Models with Diagonal Linear RNNscited by 23
  10. 2023Sparse Modular Activation for Efficient Sequence Modelingcited by 20
  11. 2021Predify: Augmenting deep neural networks with brain-inspired predictive coding dynamicscited by 19
  12. 2024State-Free Inference of State-Space Models: The Transfer Function Approachcited by 18
  13. 2021You Only Sample (Almost) Once: Linear Cost Self-Attention Via Bernoulli Samplingcited by 17
  14. 2021On-Off Center-Surround Receptive Fields for Accurate and Robust Image Classificationcited by 15
  15. 2023Recasting Self-Attention with Holographic Reduced Representationscited by 14
  16. 2023MIMONets: Multiple-Input-Multiple-Output Neural Networks Exploiting Computation in Superpositioncited by 14
  17. 2022Classification of Long Sequential Data using Circular Dilated Convolutional Neural Networkscited by 13
  18. 2024Exploring the Synergies of Hybrid CNNs and ViTs Architectures for Computer Vision: A surveycited by 13
  19. 2021Tracking Without Re-recognition in Humans and Machinescited by 12
  20. 2022Mixed Evidence for Gestalt Grouping in Deep Neural Networkscited by 12
  21. 2024From Generalization Analysis to Optimization Designs for State Space Modelscited by 11
  22. 2024Unraveling the Gradient Descent Dynamics of Transformerscited by 11
  23. 2021Predictive coding feedback results in perceived illusory contours in a recurrent neural networkcited by 10
  24. 2024Layer-Adaptive State Pruning for Deep State Space Modelscited by 10
  25. 2025Better artificial intelligence does not mean better models of biologycited by 10
  26. 2021TUNet: A Block-online Bandwidth Extension Model based on Transformers and Self-supervised Pretrainingcited by 8
  27. 2024IceFormer: Accelerated Inference with Long-Sequence Transformers on CPUscited by 8
  28. 2024Elliptical Attentioncited by 7
  29. 2021PQA: Perceptual Question Answeringcited by 6
  30. 2022ERNIE-SPARSE: Learning Hierarchical Efficient Transformer Through Regularized Self-Attentioncited by 6
  31. 2022Paramixer: Parameterizing Mixing Links in Sparse Factors Works Better than Dot-Product Self-Attentioncited by 6
  32. 2023Efficiency 360: Efficient Vision Transformerscited by 6
  33. 2023Computing a human-like reaction time metric from stable recurrent vision modelscited by 6
  34. 2023Adaptive recurrent vision performs zero-shot computation scaling to unseen difficulty levelscited by 6
  35. 2024Holographic Global Convolutional Networks for Long-Range Prediction Tasks in Malware Detectioncited by 6
  36. 2024SMR: State Memory Replay for Long Sequence Modelingcited by 6
  37. 2025Contour Integration Underlies Human-Like Visioncited by 6
  38. 2021Sketching as a Tool for Understanding and Accelerating Self-attention for Long Sequencescited by 5

50 most cited of 98 All 98 (JSON)