Bidirectional Long-Range Parser for Sequential Data Understanding
Not on the map. Why?
Abstract (arXiv)
The transformer is a powerful data modelling framework responsible for remarkable performance on a wide range of tasks. However, they are limited in terms of scalability as it is suboptimal and inefficient to process long-sequence data. To this purpose we introduce BLRP (Bidirectional Long-Range Parser), a novel and versatile attention mechanism designed to increase performance and efficiency on long-sequence tasks. It leverages short and long range heuristics in the form of a local sliding window approach combined with a global bidirectional latent space synthesis technique. We show the benefits and versatility of our approach on vision and language domains by demonstrating competitive results against state-of-the-art methods on the Long-Range-Arena and CIFAR benchmarks together with ablations demonstrating the computational efficiency.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 31 of its 42 references that are in the map. First posted on arXiv 8 Apr 2024.
Builds on
31 of 42 references.
References · 31
- 2017Attention Is All You Needcited by 59,932
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2020End-to-End Object Detection with Transformerscited by 7,384
- 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
- 2020Longformer: The Long-Document Transformercited by 3,362
- 2020Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecastingcited by 2,141
- 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
- 2020Big Bird: Transformers for Longer Sequencescited by 1,673
- 2020Rethinking Attention with Performerscited by 1,584
- 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
- 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,215
19 more
- 2022Simplified State Space Layers for Sequence Modelingcited by 776
- 2019Compressive Transformers for Long-Range Sequence Modellingcited by 699
- 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
- 2019LayoutLM: Pre-training of Text and Layout for Document Image Understandingcited by 549
- 2021Nystr"omformer: A Nystr"om-Based Algorithm for Approximating Self-Attentioncited by 468
- 2020LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understandingcited by 406
- 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
- 2015Unitary Evolution Recurrent Neural Networkscited by 291
- 2021DocFormer: End-to-End Transformer for Document Understandingcited by 254
- 2021Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encodingcited by 207
- 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 158
- 2022Liquid Structural State-Space Modelscited by 134
- 2020Lite Transformer with Long-Short Range Attentioncited by 127
- 2022How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projectionscited by 121
- 2021SelfDoc: Self-Supervised Document Representation Learningcited by 117
- 2022Block-Recurrent Transformerscited by 107
- 2021Long-Short Transformer: Efficient Transformers for Language and Visioncited by 102
- 2022Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learningcited by 13
- 2024Watermark Text Pattern Spotting in Document Imagescited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.