# PoLi-RL: A Point-to-List Reinforcement Learning Framework for Conditional Semantic Textual Similarity

arXiv: https://arxiv.org/abs/2510.04080 · HTML: https://mlsanity.com/paper/2510.04080/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods.
- Builds on 22 of its 32 references that are in the map.
- First posted on arXiv 5 Oct 2025.

## Placed in

- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 22 of 22)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [DAPO: An Open-Source LLM Reinforcement Learning System at Scale](https://mlsanity.com/paper/2503.14476/) (2025)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning](https://mlsanity.com/paper/2408.05517/) (2024)
- [UL2: Unifying Language Learning Paradigms](https://mlsanity.com/paper/2205.05131/) (2022)
- [Unsupervised Quality Estimation for Neural Machine Translation](https://mlsanity.com/paper/2005.10608/) (2020)
- [Agent KB: Leveraging Cross-Domain Experience for Agentic Problem Solving](https://mlsanity.com/paper/2507.06229/) (2025)
- [BeLLM: Backward Dependency Enhanced Large Language Model for Sentence Embeddings](https://mlsanity.com/paper/2311.05296/) (2023)
- [Simple Techniques for Enhancing Sentence Embeddings in Generative Language Models](https://mlsanity.com/paper/2404.03921/) (2024)
- [CoT-BERT: Enhancing Unsupervised Sentence Representation through Chain-of-Thought](https://mlsanity.com/paper/2309.11143/) (2023)
- [Hyper-CL: Conditioning Sentence Representations with Hypernetworks](https://mlsanity.com/paper/2403.09490/) (2024)
- [Advancing Semantic Textual Similarity Modeling: A Regression Framework with Translated ReLU and Smooth K2 Loss](https://mlsanity.com/paper/2406.05326/) (2024)
- [Pcc-tuning: Breaking the Contrastive Learning Ceiling in Semantic Textual Similarity](https://mlsanity.com/paper/2406.09790/) (2024)
- [Out-of-the-Box Conditional Text Embeddings from Large Language Models](https://mlsanity.com/paper/2504.16411/) (2025)
- [CSE-SFP: Enabling Unsupervised Sentence Representation Learning via a Single Forward Pass](https://mlsanity.com/paper/2505.00389/) (2025)
- [CoDiEmb: A Collaborative yet Distinct Framework for Unified Representation Learning in Information Retrieval and Semantic Textual Similarity](https://mlsanity.com/paper/2508.11442/) (2025)
- [Annotating Training Data for Conditional Semantic Textual Similarity Measurement using Large Language Models](https://mlsanity.com/paper/2509.14399/) (2025)

## Built on by (most cited first, 1 of 1)

- [Co-ReAct: Rubrics as Step-Level Collaborators for ReAct Agents](https://mlsanity.com/paper/2605.23590/) (2026)

## Abstract (arXiv)

Conditional Semantic Textual Similarity (C-STS) measures the semantic proximity between text segments under a specific condition, thereby overcoming the ambiguity inherent in traditional STS. However, existing methods are largely confined to discriminative models, failing to fully leverage recent breakthroughs in the NLP community involving Large Language Models (LLMs) and Reinforcement Learning (RL). RL is a particularly well-suited paradigm for this task, as it can directly optimize the non-differentiable Spearman ranking metric and guide the reasoning process required by C-STS. Nevertheless, we find that naively applying listwise RL fails to produce meaningful improvements, as the model struggles with complex, coarse-grained reward signals, leading to optimization difficulties. To address this challenge, we introduce PoLi-RL, a novel Point-to-List Reinforcement Learning framework. PoLi-RL employs a two-stage curriculum: it first trains the model with a simple pointwise reward to establish fundamental scoring capabilities, then transitions to a hybrid reward that combines pointwise, pairwise, and listwise objectives to refine the model's ability to discern subtle semantic distinctions. Crucially, we propose an innovative Parallel Slice Ranking Reward (PSRR) mechanism that computes ranking rewards in parallel slices, where each slice consists of completions with the same index from different samples. This provides a precise, differentiated learning signal for each individual completion, enabling granular credit assignment and effective optimization. On the official C-STS benchmark, PoLi-RL achieves a Spearman correlation coefficient of 48.18, establishing a new SOTA for the cross-encoder architecture. As the first work to successfully apply RL to C-STS, our study introduces a powerful paradigm for aligning LLMs for complex, ranking-based conditional judgment tasks.