# Text Encoders Lack Knowledge: Leveraging Generative LLMs for Domain-Specific Semantic Textual Similarity

arXiv: https://arxiv.org/abs/2309.06541 · HTML: https://mlsanity.com/paper/2309.06541/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 31 references that are in the map.
- First posted on arXiv 12 Sep 2023.

## Builds on (most cited first, 15 of 15)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://mlsanity.com/paper/2304.13712/) (2023)
- [Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT](https://mlsanity.com/paper/2302.10198/) (2023)
- [When do you need Chain-of-Thought Prompting for ChatGPT?](https://mlsanity.com/paper/2304.03262/) (2023)
- [Generate, Discriminate and Contrast: A Semi-Supervised Sentence Representation Learning Framework](https://mlsanity.com/paper/2210.16798/) (2022)
- [Predicting Human Similarity Judgments Using Large Language Models](https://mlsanity.com/paper/2202.04728/) (2022)

## Built on by (most cited first, 11 of 11)

- [Similarity-based Neighbor Selection for Graph LLMs](https://mlsanity.com/paper/2402.03720/) (2024)
- [SECA: Semantically Equivalent and Coherent Attacks for Eliciting LLM Hallucinations](https://mlsanity.com/paper/2510.04398/) (2025)
- ["Vorbecsti Romanecste?" A Recipe to Train Powerful Romanian LLMs with English Instructions](https://mlsanity.com/paper/2406.18266/) (2024)
- [AI Planning Framework for LLM-Based Web Agents](https://mlsanity.com/paper/2603.12710/) (2026)
- [Automated Assessment of Students' Code Comprehension using LLMs](https://mlsanity.com/paper/2401.05399/) (2023)
- [Advances and Challenges in Semantic Textual Similarity: A Comprehensive Survey](https://mlsanity.com/paper/2601.03270/) (2025)
- [Assisting humans in complex comparisons: automated information comparison at scale](https://mlsanity.com/paper/2404.04351/) (2024)
- [MIMDE: Exploring the Use of Synthetic vs Human Data for Evaluating Multi-Insight Multi-Document Extraction Tasks](https://mlsanity.com/paper/2411.19689/) (2024)
- [CrossNews-UA: A Cross-lingual News Semantic Similarity Benchmark for Ukrainian, Polish, Russian, and English](https://mlsanity.com/paper/2510.19628/) (2025)
- [Bias Testing and Mitigation in Black Box LLMs using Metamorphic Relations](https://mlsanity.com/paper/2512.00556/) (2025)
- [Feedback-to-Rubrics: Can We Learn Expert Criteria from Inline Comments?](https://mlsanity.com/paper/2605.29857/) (2026)

## Abstract (arXiv)

Amidst the sharp rise in the evaluation of large language models (LLMs) on various tasks, we find that semantic textual similarity (STS) has been under-explored. In this study, we show that STS can be cast as a text generation problem while maintaining strong performance on multiple STS benchmarks. Additionally, we show generative LLMs significantly outperform existing encoder-based STS models when characterizing the semantic similarity between two texts with complex semantic relationships dependent on world knowledge. We validate this claim by evaluating both generative LLMs and existing encoder-based STS models on three newly collected STS challenge sets which require world knowledge in the domains of Health, Politics, and Sports. All newly collected data is sourced from social media content posted after May 2023 to ensure the performance of closed-source models like ChatGPT cannot be credited to memorization. Our results show that, on average, generative LLMs outperform the best encoder-only baselines by an average of 22.3% on STS tasks requiring world knowledge. Our results suggest generative language models with STS-specific prompting strategies achieve state-of-the-art performance in complex, domain-specific STS tasks.