mlsanityData to 7 Oct 2026

Self-StrAE at SemEval-2024 Task 1: Making Self-Structuring AutoEncoders Learn More With Less

arXiv 2404.01860 ↗First posted 2 Apr 2024cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

This paper presents two simple improvements to the Self-Structuring AutoEncoder (Self-StrAE). Firstly, we show that including reconstruction to the vocabulary as an auxiliary objective improves representation quality. Secondly, we demonstrate that increasing the number of independent channels leads to significant improvements in embedding quality, while simultaneously reducing the number of parameters. Surprisingly, we demonstrate that this trend can be followed to the extreme, even to point of reducing the total number of non-embedding parameters to seven. Our system can be pre-trained from scratch with as little as 10M tokens of input data, and proves effective across English, Spanish and Afrikaans.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 15 of its 24 references that are in the map. First posted on arXiv 2 Apr 2024.

Builds on

15 of 24 references.

References · 15

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  7. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  8. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  9. 2014SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimationcited by 170
  10. 2021Can contrastive learning avoid shortcut solutions?cited by 114
  11. 2019Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Autoencoderscited by 60
  12. 2017BPEmb: Tokenization-free Pre-trained Subword Embeddings in 275 Languagescited by 52
3 more
  1. 2020Relating by Contrasting: A Data-efficient Framework for Multimodal Generative Modelscited by 27
  2. 2021R2D2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modelingcited by 19
  3. 2024SemEval-2024 Task 1: Semantic Textual Relatedness for African and Asian Languagescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 1.

3 papers.

  1. 2024Banyan: Improved Representation Learning with Explicit Structurecited by 3
  2. 2025TRA: Better Length Generalisation with Threshold Relative Attentioncited by 1
  3. 2026Bootstrapping Embeddings for Low Resource Languagescited by 0