# ElicitationGPT: Text Elicitation Mechanisms via Language Models

arXiv: https://arxiv.org/abs/2406.09363 · HTML: https://mlsanity.com/paper/2406.09363/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 28 references that are in the map.
- First posted on arXiv 13 Jun 2024.

## Builds on (most cited first, 14 of 14)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [From Arguments to Key Points: Towards Automatic Argument Summarization](https://mlsanity.com/paper/2005.01619/) (2020)
- [When Does Optimizing a Proper Loss Yield Calibration?](https://mlsanity.com/paper/2305.18764/) (2023)
- [Optimization of Scoring Rules](https://mlsanity.com/paper/2007.02905/) (2020)
- [Towards LLM-based Autograding for Short Textual Answers](https://mlsanity.com/paper/2309.11508/) (2023)
- [Loss Minimization Yields Multicalibration for Large Neural Networks](https://mlsanity.com/paper/2304.09424/) (2023)
- [Eliciting Informative Text Evaluations with Large Language Models](https://mlsanity.com/paper/2405.15077/) (2024)
- [Optimal Scoring Rules for Multi-dimensional Effort](https://mlsanity.com/paper/2211.03302/) (2022)
- [Eliciting Expertise without Verification](https://mlsanity.com/paper/1802.08312/) (2018)
- [Contracts with Information Acquisition, via Scoring Rules](https://mlsanity.com/paper/2204.01773/) (2022)
- [Proper losses for discrete generative models](https://mlsanity.com/paper/2211.03761/) (2022)

## Built on by (most cited first, 9 of 9)

- [Causal Inference on Outcomes Learned from Text](https://mlsanity.com/paper/2503.00725/) (2025)
- [Truthful Elicitation of Imprecise Forecasts](https://mlsanity.com/paper/2503.16395/) (2025)
- [Evaluating LLM-Contaminated Crowdsourcing Data Without Ground Truth](https://mlsanity.com/paper/2506.06991/) (2025)
- [Aligned Textual Scoring Rules](https://mlsanity.com/paper/2507.06221/) (2025)
- [Incentive-Aligned Multi-Source LLM Summaries](https://mlsanity.com/paper/2509.25184/) (2025)
- [Let's Measure Information Step-by-Step: AI-Based Evaluation Beyond Vibes](https://mlsanity.com/paper/2508.05469/) (2025)
- [Correct Looks Better: Pairwise Comparisons Reveal Accuracy Rankings](https://mlsanity.com/paper/2606.09409/) (2026)
- [Scoring Rules! Statistical and Strategic Alignment for Text Evaluation Metrics](https://mlsanity.com/paper/2608.01423/) (2026)
- [How the Audit Rule Shapes Faithful Factor Explanations in LLMs](https://mlsanity.com/paper/2610.01514/) (2026)

## Abstract (arXiv)

Scoring rules evaluate probabilistic forecasts of an unknown state against the realized state and are a fundamental building block in the incentivized elicitation of information. This paper develops mechanisms for scoring elicited text against ground truth text by reducing the textual information elicitation problem to a forecast elicitation problem, via domain-knowledge-free queries to a large language model (specifically ChatGPT), and empirically evaluates their alignment with human preferences. Our theoretical analysis shows that the reduction achieves provable properness via black-box language models. The empirical evaluation is conducted on peer reviews from a peer-grading dataset, in comparison to manual instructor scores for the peer reviews. Our results suggest a paradigm of algorithmic artificial intelligence that may be useful for developing artificial intelligence technologies with provable guarantees.