# SemCSE: Semantic Contrastive Sentence Embeddings Using LLM-Generated Summaries For Scientific Abstracts

arXiv: https://arxiv.org/abs/2507.13105 · HTML: https://mlsanity.com/paper/2507.13105/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 44 references that are in the map.
- First posted on arXiv 17 Jul 2025.

## Builds on (most cited first, 28 of 28)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://mlsanity.com/paper/2405.17428/) (2024)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [Deep metric learning using Triplet network](https://mlsanity.com/paper/1412.6622/) (2014)
- [Representation Degeneration Problem in Training Natural Language Generation Models](https://mlsanity.com/paper/1907.12009/) (2019)
- [CLEAR: Contrastive Learning for Sentence Representation](https://mlsanity.com/paper/2012.15466/) (2020)
- [jina-embeddings-v3: Multilingual Embeddings With Task LoRA](https://mlsanity.com/paper/2409.10173/) (2024)
- [SciRepEval: A Multi-Format Benchmark for Scientific Document Representations](https://mlsanity.com/paper/2211.13308/) (2022)
- [Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documents](https://mlsanity.com/paper/2310.19923/) (2023)
- [Growth rates of modern science: A latent piecewise growth curve approach to model publication numbers from established and new literature databases](https://mlsanity.com/paper/2012.07675/) (2020)
- [Neighborhood Contrastive Learning for Scientific Document Representations with Citation Embeddings](https://mlsanity.com/paper/2202.06671/) (2022)
- [WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach](https://mlsanity.com/paper/2104.01767/) (2021)
- [Content-Based Citation Recommendation](https://mlsanity.com/paper/1802.08301/) (2018)
- [Multi-Vector Models with Textual Guidance for Fine-Grained Scientific Document Similarity](https://mlsanity.com/paper/2111.08366/) (2021)
- [GenEOL: Harnessing the Generative Power of LLMs for Training-Free Sentence Embeddings](https://mlsanity.com/paper/2410.14635/) (2024)
- [Improving Document Representations by Generating Pseudo Query Embeddings for Dense Retrieval](https://mlsanity.com/paper/2105.03599/) (2021)
- [Evaluating Small Language Models for News Summarization: Implications and Factors Influencing Performance](https://mlsanity.com/paper/2502.00641/) (2025)
- [Little Giants: Synthesizing High-Quality Embedding Data at Scale](https://mlsanity.com/paper/2410.18634/) (2024)
- [A Scalable Hybrid Research Paper Recommender System for Microsoft Academic](https://mlsanity.com/paper/1905.08880/) (2019)
- [Science Concierge: A fast content-based recommendation system for scientific publications](https://mlsanity.com/paper/1604.01070/) (2016)
- [Enhancing Domain-Specific Encoder Models with LLM-Generated Data: How to Leverage Ontologies, and How to Do Without Them](https://mlsanity.com/paper/2503.22006/) (2025)
- [Document Embedding for Scientific Articles: Efficacy of Word Embeddings vs TFIDF](https://mlsanity.com/paper/2107.05151/) (2021)
- [Preserving Multilingual Quality While Tuning Query Encoder on English Only](https://mlsanity.com/paper/2407.00923/) (2024)

## Built on by (most cited first, 4 of 4)

- [ChEmbed: Enhancing Chemical Literature Search Through Domain-Specific Text Embeddings](https://mlsanity.com/paper/2508.01643/) (2025)
- [SemCSE-Multi: Multifaceted and Decodable Embeddings for Aspect-Specific and Interpretable Scientific Domain Mapping](https://mlsanity.com/paper/2510.11599/) (2025)
- [Asymmetric Within-Document Predictive Learning for Scientific Document Representation](https://mlsanity.com/paper/2608.28625/) (2026)
- [Same Problem, Different Field: Cross-Domain Solution Import via Domain-Stripped Computational Fingerprints](https://mlsanity.com/paper/2609.07595/) (2026)

## Abstract (arXiv)

We introduce SemCSE, an unsupervised method for learning semantic embeddings of scientific texts. Building on recent advances in contrastive learning for text embeddings, our approach leverages LLM-generated summaries of scientific abstracts to train a model that positions semantically related summaries closer together in the embedding space. This resulting objective ensures that the model captures the true semantic content of a text, in contrast to traditional citation-based approaches that do not necessarily reflect semantic similarity. To validate this, we propose a novel benchmark designed to assess a model's ability to understand and encode the semantic content of scientific texts, demonstrating that our method enforces a stronger semantic separation within the embedding space. Additionally, we evaluate SemCSE on the comprehensive SciRepEval benchmark for scientific text embeddings, where it achieves state-of-the-art performance among models of its size, thus highlighting the benefits of a semantically focused training approach.