mlsanityData to 14 Oct 2026

Know Yourself and Know Your Neighbour: A Syntactically Informed Self-Supervised Compositional Sentence Representation Learning Framework using a Recursive Hypernetwork

arXiv 2610.16547 ↗First posted 12 Oct 2026cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Sentence representation learning is still an open challenge in Natural Language Processing. In this work, we propose a new self-supervised framework for learning sentence representations, using a special type of neural network called a recursive hypernetwork. Our proposed model composes the representation of a sentence from representations of words by applying a recursive composition through the parse tree. We maintain a separate syntactic and semantic representation, and the semantic composition is guided by the information from the syntactic representation. To train this model, we introduce a novel set of six self-supervised tasks. By analysing the performance on 7 probing tasks, we validate that the generated sentence representation encodes richer linguistic information than both averaging baselines and state-of-the-art alternatives. Furthermore, we assess the impact of the six proposed self-supervised training tasks through ablation studies. We also demonstrate that the representations generated by our model are stable for sentences of varying length and that the semantic composition operators adapt to different syntactic categoriesfootnoteCode available: https://github.com/vasudev2020/codi/tree/main.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 48 of its 70 references that are in the map. First posted on arXiv 12 Oct 2026.

Builds on

48 of 70 references.

References · 48

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2016Layer Normalizationcited by 5,669
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  7. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  8. 2014How transferable are features in deep neural networks?cited by 1,225
  9. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  10. 2014Distributed Representations of Sentences and Documentscited by 678
  11. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  12. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 457
36 more
  1. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 432
  2. 2015Skip-Thought Vectorscited by 322
  3. 2015Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networkscited by 306
  4. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 251
  5. 2019Visualizing and Measuring the Geometry of BERTcited by 216
  6. 2017All-but-the-Top: Simple and Effective Postprocessing for Word Representationscited by 212
  7. 2018Constituency Parsing with a Self-Attentive Encodercited by 199
  8. 2018An efficient framework for learning sentence representationscited by 198
  9. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 177
  10. 2019Open Sesame: Getting Inside BERT's Linguistic Knowledgecited by 120
  11. 2019StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understandingcited by 109
  12. 2018Auxiliary Tasks in Multi-task Learningcited by 108
  13. 2017Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Featurescited by 104
  14. 2016Learning Distributed Representations of Sentences from Unlabelled Datacited by 91
  15. 2020Syntactic Data Augmentation Increases Robustness to Inference Heuristicscited by 86
  16. 2020Out of Order: How Important Is The Sequential Order of Words in a Sentence in Natural Language Understanding Tasks?cited by 73
  17. 2020UnNatural Language Inferencecited by 69
  18. 2015Towards Universal Paraphrastic Sentence Embeddingscited by 60
  19. 2020SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Modelscited by 56
  20. 2021IsoScore: Measuring the Uniformity of Embedding Space Utilizationcited by 53
  21. 2020Do Syntax Trees Help Pre-trained Transformers Extract Information?cited by 48
  22. 2022Characterizing Intrinsic Compositionality in Transformers with Tree Projectionscited by 41
  23. 2020Syntax-Enhanced Pre-trained Modelcited by 30
  24. 2017Jointly Learning Sentence Embeddings and Syntax with Unsupervised Tree-LSTMscited by 24
  25. 2022Fast-R2D2: A Pretrained Recursive Neural Network based on Pruned CKY for Grammar Induction and Text Representationcited by 15
  26. 2022Probing for Constituency Structure in Neural Language Modelscited by 14
  27. 2023A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the ChatGPT Era and Beyondcited by 12
  28. 2015Compositional Distributional Semantics with Long Short Term Memorycited by 10
  29. 2018Dynamic Compositionality in Recursive Neural Networks with Structure-aware Tag Representationscited by 6
  30. 2019Improving Natural Language Inference with a Pretrained Parsercited by 5
  31. 2015Long Short-Term Memory Over Tree Structurescited by 4
  32. 2018Learning to Compose over Tree Structures via POS Tagscited by 4
  33. 2019HUBERT Untangles BERT to Improve Transfer across NLP Taskscited by 4
  34. 2017Dynamic Compositional Neural Networks over Tree Structurecited by 3
  35. 2024Topic Aware Probing: From Sentence Length Prediction to Idiom Identification how reliant are Neural Language Models on Topic?cited by 3
  36. 2024Leveraging Grammar Induction for Language Understanding and Generationcited by 1

Built on by

Not cited yet.