mlsanityData to 14 Oct 2026

BeLLM: Backward Dependency Enhanced Large Language Model for Sentence Embeddings

arXiv 2311.05296 ↗First posted 9 Nov 2023cs.CLcited by 32

Not on the map. Why?

Abstract (arXiv)

Sentence embeddings are crucial in measuring semantic similarity. Most recent studies employed large language models (LLMs) to learn sentence embeddings. Existing LLMs mainly adopted autoregressive architecture without explicit backward dependency modeling. Therefore, we examined the effects of backward dependencies in LLMs for semantic similarity measurements. Concretely, we propose a novel model: backward dependency enhanced large language model (BeLLM). It learns sentence embeddings via transforming specific attention layers from uni- to bi-directional. We extensively experiment across various semantic textual similarity (STS) tasks and downstream applications. BeLLM achieves state-of-the-art performance in varying scenarios. It shows that auto-regressive LLMs benefit from backward dependencies for sentence embeddings.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Builds on 26 of its 61 references that are in the map. First posted on arXiv 9 Nov 2023.

Builds on

26 of 61 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  11. 2017Billion-scale similarity search with GPUscited by 2,497
  12. 2018Deep contextualized word representationscited by 2,290
14 more
  1. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  2. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  3. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  4. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  5. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  6. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 784
  7. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 432
  8. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  9. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  10. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 202
  11. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 132
  12. 2020An Unsupervised Sentence Embedding Method by Mutual Information Maximizationcited by 91
  13. 2023Whitening-based Contrastive Learning of Sentence Embeddingscited by 21
  14. 2023HiCL: Hierarchical Contrastive Learning of Unsupervised Sentence Embeddingscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 15; 2026: 7.

32 papers.

Sort:

By area:

  1. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 404
  2. 2024Generative Representational Instruction Tuningcited by 233
  3. 2024Token Prepending: A Training-Free Approach for Eliciting Better Sentence Embeddings from LLMscited by 18
  4. 2024Meta-Task Prompting Elicits Embeddings from Large Language Modelscited by 16
  5. 2024GenEOL: Harnessing the Generative Power of LLMs for Training-Free Sentence Embeddingscited by 15
  6. 2025Contrastive Prompting Enhances Sentence Embeddings in LLMs through Inference-Time Steeringcited by 13
  7. 2025ProRank: Prompt Warmup via Reinforcement Learning for Small Language Models Rerankingcited by 11
  8. 2025Read Before You Think: Mitigating LLM Comprehension Failures with Step-by-Step Readingcited by 10
  9. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  10. 2024Large Language Models Are Overparameterized Text Encoderscited by 6
  11. 2024From Word Vectors to Multimodal Embeddings: Techniques, Applications, and Future Directions For Large Language Modelscited by 6
  12. 2025MAGNET: Augmenting Generative Decoders with Representation Learning and Infilling Capabilitiescited by 5
20 more
  1. 2025New Encoders for German Trained from Scratch: Comparing ModernGBERT with Converted LLM2Vec Modelscited by 5
  2. 2024Enhancing Unsupervised Sentence Embeddings via Knowledge-Driven Data Augmentation and Gaussian-Decayed Contrastive Learningcited by 4
  3. 2025RegionMarker: A Region-Triggered Semantic Watermarking Framework for Embedding-as-a-Service Copyright Protectioncited by 4
  4. 2025Hierarchical Token Prepending: Enhancing Information Flow in Decoder-based LLM Embeddingscited by 3
  5. 2024Generative Deduplication For Socia Media Data Selectioncited by 2
  6. 2025ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Modelscited by 2
  7. 2025Educational Cone Model in Embedding Vector Spacescited by 2
  8. 2026Analyzing Diffusion and Autoregressive Vision Language Models in Multimodal Embedding Spacecited by 2
  9. 2024Privately Learning from Graphs with Applications in Fine-tuning Large Language Modelscited by 1
  10. 2025CASE -- Condition-Aware Sentence Embeddings for Conditional Semantic Textual Similarity Measurementcited by 1
  11. 2025PoLi-RL: A Point-to-List Reinforcement Learning Framework for Conditional Semantic Textual Similaritycited by 1
  12. 2026Focusing Condition: Inference-Time Self-Contrastive Steering Elicits Better Conditional Text Embeddings in LLMscited by 1
  13. 2025Causal2Vec: Improving Decoder-only LLMs as Embedding Models through a Contextual Tokencited by 0
  14. 2025Causal Attention with Lookahead Keyscited by 0
  15. 2025Text2Token: Unsupervised Text Representation Learning with Token Target Predictioncited by 0
  16. 2026Geometry-Aware Localized Watermarking for Copyright Protection in Embedding-as-a-Servicecited by 0
  17. 2026Modular Representation Compression: Adapting LLMs for Efficient and Effective Recommendationscited by 0
  18. 2026Embedding-based In-Context Prompt Training for Enhancing LLMs as Text Encoderscited by 0
  19. 2026ReverseEOL: Improving Training-free Text Embeddings via Text Reversal in Decoder-only LLMscited by 0
  20. 2026Selecting What Matters: Semantic Compression-Guided Selective Pooling for Long-Context Embeddingscited by 0