mlsanityData to 7 Oct 2026

AnglE-optimized Text Embeddings

arXiv 2309.12871 ↗First posted 22 Sep 2023cs.CL cs.AI cs.LGcited by 98

Not on the map. Why?

Abstract (arXiv)

High-quality text embedding is pivotal in improving semantic textual similarity (STS) tasks, which are crucial components in Large Language Model (LLM) applications. However, a common challenge existing text embedding models face is the problem of vanishing gradients, primarily due to their reliance on the cosine function in the optimization objective, which has saturation zones. To address this issue, this paper proposes a novel angle-optimized text embedding model called AnglE. The core idea of AnglE is to introduce angle optimization in a complex space. This novel approach effectively mitigates the adverse effects of the saturation zone in the cosine function, which can impede gradient and hinder optimization processes. To set up a comprehensive STS evaluation, we experimented on existing short-text STS datasets and a newly collected long-text STS dataset from GitHub Issues. Furthermore, we examine domain-specific STS scenarios with limited labeled data and explore how AnglE works with LLM-annotated data. Extensive experiments were conducted on various tasks including short-text STS, long-text STS, and domain-specific STS tasks. The results show that AnglE outperforms the state-of-the-art (SOTA) STS models that ignore the cosine saturation zone. These findings demonstrate the ability of AnglE to generate high-quality text embeddings and the usefulness of angle optimization in STS.

Key facts. Cited by 98 papers in the map (data through 7 Oct 2026). Builds on 33 of its 52 references that are in the map. First posted on arXiv 22 Sep 2023.

Builds on

33 of 52 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  8. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  9. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  11. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,837
21 more
  1. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  2. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  3. 2019RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Spacecited by 893
  4. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  5. 2015Skip-Thought Vectorscited by 321
  6. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  7. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  8. 2021ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfercited by 231
  9. 2021Whitening Sentence Representations for Better Semantics and Faster Retrievalcited by 198
  10. 2022DiffCSE: Difference-based Contrastive Learning for Sentence Embeddingscited by 131
  11. 2017Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Featurescited by 103
  12. 2020An Unsupervised Sentence Embedding Method by Mutual Information Maximizationcited by 91
  13. 2016Learning Distributed Representations of Sentences from Unlabelled Datacited by 90
  14. 2023Scaling Sentence Embeddings with Large Language Modelscited by 83
  15. 2022PromptBERT: Improving BERT Sentence Embeddings with Promptscited by 75
  16. 2021Not All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text Classificationcited by 43
  17. 2022Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learningcited by 30
  18. 2023Whitening-based Contrastive Learning of Sentence Embeddingscited by 21
  19. 2021Contrastive Domain Adaptation for Question Answering using Limited Text Corporacited by 20
  20. 2023Composition-contrastive Learning for Sentence Embeddingscited by 14
  21. 2023Recurrent Attention Networks for Long-text Modelingcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 56; 2025: 26; 2026: 11.

98 papers.

Sort:

By area:

  1. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  2. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 466
  3. 2023Improving Text Embeddings with Large Language Modelscited by 295
  4. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 251
  5. 2024Generative Representational Instruction Tuningcited by 227
  6. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  7. 2024Repetition Improves Language Model Embeddingscited by 70
  8. 2024FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructionscited by 67
  9. 2024KAG: Boosting LLMs in Professional Domains via Knowledge Augmented Generationcited by 67
  10. 2024TrojanRAG: Retrieval-Augmented Generation Can Be Backdoor Driver in Large Language Modelscited by 63
  11. 2024DeTeCtive: Detecting AI-generated Text via Multi-Level Contrastive Learningcited by 52
  12. 2024Multicalibration for Confidence Scoring in LLMscited by 44
38 more
  1. 2024SUGARCREPE++ Dataset: Vision-Language Model Sensitivity to Semantic and Lexical Alterationscited by 42
  2. 2024GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuningcited by 40
  3. 2024BEAVER: An Enterprise Benchmark for Text-to-SQLcited by 37
  4. 2025Towards Label-Only Membership Inference Attack against Pre-trained Large Language Modelscited by 34
  5. 2024BERGEN: A Benchmarking Library for Retrieval-Augmented Generationcited by 31
  6. 2024A Semantic Search Engine for Mathlib4cited by 30
  7. 2024Simple Techniques for Enhancing Sentence Embeddings in Generative Language Modelscited by 26
  8. 2023Label Supervised LLaMA Finetuningcited by 20
  9. 2024Aloe: A Family of Fine-tuned Open Healthcare LLMscited by 20
  10. 2024FeB4RAG: Evaluating Federated Search in the Context of Retrieval Augmented Generationcited by 19
  11. 2024Recent advances in text embedding: A Comprehensive Review of Top-Performing Methods on the MTEB Benchmarkcited by 19
  12. 2024Cost-Effective Hallucination Detection for LLMscited by 16
  13. 2024ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentationcited by 16
  14. 2025EnrichIndex: Using LLMs to Enrich Retrieval Indices Offlinecited by 15
  15. 2025NeoBERT: A Next-Generation BERTcited by 14
  16. 2024Rethinking Visual Prompting for Multimodal Large Language Models with External Knowledgecited by 13
  17. 2024Generative Adversarial Reviews: When LLMs Become the Criticcited by 12
  18. 2025Retrieval-augmented Large Language Models for Financial Time Series Forecastingcited by 12
  19. 2025Who Taught the Lie? Responsibility Attribution for Poisoned Knowledge in Retrieval-Augmented Generationcited by 12
  20. 2024Learning on Graphs with Large Language Models(LLMs): A Deep Dive into Model Robustnesscited by 11
  21. 2024Automated Knowledge Concept Annotation and Question Representation Learning for Knowledge Tracingcited by 11
  22. 2025Defending Against Knowledge Poisoning Attacks During Retrieval-Augmented Generationcited by 11
  23. 2025Large language models as uncertainty-calibrated optimizers for experimental discoverycited by 10
  24. 2024Language-Conditioned Offline RL for Multi-Robot Navigationcited by 9
  25. 2024Extracting Paragraphs from LLM Token Activationscited by 9
  26. 2025SAGE: A Framework of Precise Retrieval for RAGcited by 8
  27. 2025A Survey on Retrieval And Structuring Augmented Generation with Large Language Modelscited by 8
  28. 2024PersianMind: A Cross-Lingual Persian-English Large Language Modelcited by 7
  29. 2025HUME: Measuring the Human-Model Performance Gap in Text Embedding Taskscited by 7
  30. 2024DocTabQA: Answering Questions from Long Documents Using Tablescited by 6
  31. 2024CIPHER: Cybersecurity Intelligent Penetration-testing Helper for Ethical Researchercited by 6
  32. 2024Thematic Analysis with Open-Source Generative AI and Machine Learning: A New Method for Inductive Qualitative Codebook Developmentcited by 6
  33. 2024Towards Long Video Understanding via Fine-detailed Video Story Generationcited by 6
  34. 2025VIBE: Vector Index Benchmark for Embeddingscited by 6
  35. 2024TAGA: Text-Attributed Graph Self-Supervised Learning by Synergizing Graph and Text Mutual Transformationscited by 5
  36. 2024Towards Unlocking Insights from Logbooks Using AIcited by 5
  37. 2025LexSemBridge: Fine-Grained Dense Representation Enhancement through Token-Aware Embedding Augmentationcited by 5
  38. 2024Using Generative Text Models to Create Qualitative Codebooks for Student Evaluations of Teachingcited by 4

50 most cited of 98 All 98 (JSON)