mlsanityData to 14 Oct 2026

ElicitationGPT: Text Elicitation Mechanisms via Language Models

arXiv 2406.09363 ↗First posted 13 Jun 2024cs.AI cs.GT cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Scoring rules evaluate probabilistic forecasts of an unknown state against the realized state and are a fundamental building block in the incentivized elicitation of information. This paper develops mechanisms for scoring elicited text against ground truth text by reducing the textual information elicitation problem to a forecast elicitation problem, via domain-knowledge-free queries to a large language model (specifically ChatGPT), and empirically evaluates their alignment with human preferences. Our theoretical analysis shows that the reduction achieves provable properness via black-box language models. The empirical evaluation is conducted on peer reviews from a peer-grading dataset, in comparison to manual instructor scores for the peer reviews. Our results suggest a paradigm of algorithmic artificial intelligence that may be useful for developing artificial intelligence technologies with provable guarantees.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 14 of its 28 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

14 of 28 references.

References · 14

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  4. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  5. 2020From Arguments to Key Points: Towards Automatic Argument Summarizationcited by 61
  6. 2023When Does Optimizing a Proper Loss Yield Calibration?cited by 36
  7. 2020Optimization of Scoring Rulescited by 28
  8. 2023Towards LLM-based Autograding for Short Textual Answerscited by 20
  9. 2023Loss Minimization Yields Multicalibration for Large Neural Networkscited by 19
  10. 2024Eliciting Informative Text Evaluations with Large Language Modelscited by 15
  11. 2022Optimal Scoring Rules for Multi-dimensional Effortcited by 12
  12. 2018Eliciting Expertise without Verificationcited by 11
2 more
  1. 2022Contracts with Information Acquisition, via Scoring Rulescited by 8
  2. 2022Proper losses for discrete generative modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 3.

9 papers.

  1. 2025Causal Inference on Outcomes Learned from Textcited by 8
  2. 2025Truthful Elicitation of Imprecise Forecastscited by 6
  3. 2025Evaluating LLM-Contaminated Crowdsourcing Data Without Ground Truthcited by 2
  4. 2025Aligned Textual Scoring Rulescited by 2
  5. 2025Incentive-Aligned Multi-Source LLM Summariescited by 1
  6. 2025Let's Measure Information Step-by-Step: AI-Based Evaluation Beyond Vibescited by 0
  7. 2026Correct Looks Better: Pairwise Comparisons Reveal Accuracy Rankingscited by 0
  8. 2026Scoring Rules! Statistical and Strategic Alignment for Text Evaluation Metricscited by 0
  9. 2026How the Audit Rule Shapes Faithful Factor Explanations in LLMscited by 0