mlsanityData to 7 Oct 2026

Banyan: Improved Representation Learning with Explicit Structure

arXiv 2407.17771 ↗First posted 25 Jul 2024cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

We present Banyan, a model that efficiently learns semantic representations by leveraging explicit hierarchical structure. While transformers excel at scale, they struggle in low-resource settings. Conversely recent structured models have shown promise as efficient learners, but lack performance. Banyan bridges this gap with two key innovations: an entangled hierarchical tree structure and diagonalized message passing, enabling it to outperform larger transformer models with just 14 non-embedding parameters. It excels in low-resource settings, offering a viable alternative for under-represented languages and highlighting its potential for efficient, interpretable NLP in resource-constrained environments.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 41 of its 76 references that are in the map. First posted on arXiv 25 Jul 2024.

Builds on

41 of 76 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
  8. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  9. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  10. 2016Pointer Sentinel Mixture Modelscited by 2,992
  11. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  12. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
29 more
  1. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  2. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  3. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  4. 2015Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networkscited by 305
  5. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  6. 2014SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimationcited by 170
  7. 2016Using Fast Weights to Attend to the Recent Pastcited by 161
  8. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 128
  9. 2019Well-Read Students Learn Better: On the Importance of Pre-training Compact Modelscited by 118
  10. 2021Can contrastive learning avoid shortcut solutions?cited by 114
  11. 2017Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Featurescited by 103
  12. 2024Anisotropy Is Inherent to Self-Attention in Transformerscited by 62
  13. 2019Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Autoencoderscited by 60
  14. 2017BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languagescited by 52
  15. 2022Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scalecited by 40
  16. 2019Cooperative Learning of Disjoint Syntax and Semanticscited by 27
  17. 2022Compositional generalization through abstract representations in human and artificial neural networkscited by 21
  18. 2022Just Rank: Rethinking Evaluation with Word and Sentence Similaritiescited by 20
  19. 2021Paraphrastic Representations at Scalecited by 19
  20. 2021R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modelingcited by 19
  21. 2023Pushdown Layers: Encoding Recursive Structure in Transformer Language Modelscited by 18
  22. 2024SemRel2024: A Collection of Semantic Textual Relatedness Datasets for 13 Languagescited by 16
  23. 2021Modeling Hierarchical Structures with Continuous Recursive Neural Networkscited by 15
  24. 2021What Makes Sentences Semantically Related: A Textual Relatedness Dataset and Empirical Studycited by 15
  25. 2022Fast-R2D2: A Pretrained Recursive Neural Network based on Pruned CKY for Grammar Induction and Text Representationcited by 14
  26. 2024SemEval-2024 Task 1: Semantic Textual Relatedness for African and Asian Languagescited by 7
  27. 2023Beam Tree Recursive Cellscited by 6
  28. 2023On the effect of curriculum learning with developmental data for grammar acquisitioncited by 5
  29. 2024Self-StrAE at SemEval-2024 Task 1: Making Self-Structuring AutoEncoders Learn More With Lesscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025TRA: Better Length Generalisation with Threshold Relative Attentioncited by 1
  2. 2026Bootstrapping Embeddings for Low Resource Languagescited by 0
  3. 2026Hierarchical ODE: Learning Continuous-Time Physical Prototypes for Early Link Failure Detectioncited by 0