# Unsupervised Contrast-Consistent Ranking with Language Models

arXiv: https://arxiv.org/abs/2309.06991 · HTML: https://mlsanity.com/paper/2309.06991/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 52 references that are in the map.
- First posted on arXiv 13 Sep 2023.

## Builds on (most cited first, 26 of 26)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Large Language Models are not Fair Evaluators](https://mlsanity.com/paper/2305.17926/) (2023)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [Contrastive Decoding: Open-ended Text Generation as Optimization](https://mlsanity.com/paper/2210.15097/) (2022)
- [How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering](https://mlsanity.com/paper/2012.00955/) (2020)
- [Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting](https://mlsanity.com/paper/2306.17563/) (2023)
- [What do Neural Machine Translation Models Learn about Morphology?](https://mlsanity.com/paper/1704.03471/) (2017)
- [Rank consistent ordinal regression for neural networks with application to age estimation](https://mlsanity.com/paper/1901.07884/) (2019)
- [A Theoretical Analysis of NDCG Type Ranking Measures](https://mlsanity.com/paper/1304.6480/) (2013)
- [Zero-Shot Listwise Document Reranking with a Large Language Model](https://mlsanity.com/paper/2305.02156/) (2023)
- [Deep Neural Networks for Rank-Consistent Ordinal Regression Based On Conditional Probabilities](https://mlsanity.com/paper/2111.08851/) (2021)
- [Prototypical Calibration for Few-shot Learning of Language Models](https://mlsanity.com/paper/2205.10183/) (2022)
- [BERT Knows Punta Cana is not just beautiful, it's gorgeous: Ranking Scalar Adjectives with Contextualised Representations](https://mlsanity.com/paper/2010.02686/) (2020)
- [Discovering Differences in the Representation of People using Contextualized Semantic Axes](https://mlsanity.com/paper/2210.12170/) (2022)
- [SensePOLAR: Word sense aware interpretability for pre-trained contextual word embeddings](https://mlsanity.com/paper/2301.04704/) (2023)
- [Prompt-based Learning for Text Readability Assessment](https://mlsanity.com/paper/2302.13139/) (2023)
- [Language Modelling via Learning to Rank](https://mlsanity.com/paper/2110.06961/) (2021)
- [Probing via Prompting](https://mlsanity.com/paper/2207.01736/) (2022)
- [Exploring Contrast Consistency of Open-Domain Question Answering Systems on Minimally Edited Questions](https://mlsanity.com/paper/2305.14441/) (2023)
- [The Ordered Matrix Dirichlet for State-Space Models](https://mlsanity.com/paper/2212.04130/) (2022)

## Built on by (most cited first, 13 of 13)

- [Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Models](https://mlsanity.com/paper/2310.07712/) (2023)
- [Localizing Paragraph Memorization in Language Models](https://mlsanity.com/paper/2403.19851/) (2024)
- [Context versus Prior Knowledge in Language Models](https://mlsanity.com/paper/2404.04633/) (2024)
- [Self-Recognition in Language Models](https://mlsanity.com/paper/2407.06946/) (2024)
- [Measurement in the Age of LLMs: An Application to Ideological Scaling](https://mlsanity.com/paper/2312.09203/) (2023)
- [What Do Llamas Really Think? Revealing Preference Biases in Language Model Representations](https://mlsanity.com/paper/2311.18812/) (2023)
- [On Fairness of Low-Rank Adaptation of Large Models](https://mlsanity.com/paper/2405.17512/) (2024)
- [Measuring Scalar Constructs in Social Science with LLMs](https://mlsanity.com/paper/2509.03116/) (2025)
- [Self-Calibrated Listwise Reranking with Large Language Models](https://mlsanity.com/paper/2411.04602/) (2024)
- [LayerCake: Token-Aware Contrastive Decoding within Large Language Model Layers](https://mlsanity.com/paper/2507.04404/) (2025)
- [Polarity-Aware Probing for Quantifying Latent Alignment in Language Models](https://mlsanity.com/paper/2511.21737/) (2025)
- [Mind the Shift: Decoding Monetary Policy Stance from FOMC Statements with Large Language Models](https://mlsanity.com/paper/2603.14313/) (2026)
- [Probing Ethical Framework Representations in Large Language Models: Structure, Entanglement, and Methodological Challenges](https://mlsanity.com/paper/2603.23659/) (2026)

## Abstract (arXiv)

Language models contain ranking-based knowledge and are powerful solvers of in-context ranking tasks. For instance, they may have parametric knowledge about the ordering of countries by size or may be able to rank product reviews by sentiment. We compare pairwise, pointwise and listwise prompting techniques to elicit a language model's ranking knowledge. However, we find that even with careful calibration and constrained decoding, prompting-based techniques may not always be self-consistent in the rankings they produce. This motivates us to explore an alternative approach that is inspired by an unsupervised probing method called Contrast-Consistent Search (CCS). The idea is to train a probe guided by a logical constraint: a language model's representation of a statement and its negation must be mapped to contrastive true-false poles consistently across multiple statements. We hypothesize that similar constraints apply to ranking tasks where all items are related via consistent, pairwise or listwise comparisons. To this end, we extend the binary CCS method to Contrast-Consistent Ranking (CCR) by adapting existing ranking methods such as the Max-Margin Loss, Triplet Loss and an Ordinal Regression objective. Across different models and datasets, our results confirm that CCR probing performs better or, at least, on a par with prompting.