mlsanityData to 14 Oct 2026

RELIC: Investigating Large Language Model Responses using Self-Consistency

arXiv 2311.16842 ↗First posted 28 Nov 2023cs.HC cs.CLcited by 23

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Large Language Models (LLMs) are notorious for blending fact with fiction and generating non-factual content, known as hallucinations. To address this challenge, we propose an interactive system that helps users gain insight into the reliability of the generated text. Our approach is based on the idea that the self-consistency of multiple samples generated by the same LLM relates to its confidence in individual claims in the generated texts. Using this idea, we design RELIC, an interactive system that enables users to investigate and verify semantic-level variations in multiple long-form responses. This allows users to recognize potentially inaccurate information in the generated text and make necessary corrections. From a user study with ten participants, we demonstrate that our approach helps users better verify the reliability of the generated text. We further summarize the design implications and lessons learned from this research for future studies of reliable human-LLM interactions.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 43 of its 69 references that are in the map. First posted on arXiv 28 Nov 2023.

Builds on

43 of 69 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing19 May
  2. 2022Language Models (Mostly) Know What They Know11 Jul

Other references · 41

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  6. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  7. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  8. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  9. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  10. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  11. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 458
  12. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
29 more
  1. 2020Effect of Confidence and Explanation on Accuracy and Trust Calibration in AI-Assisted Decision Makingcited by 281
  2. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 268
  3. 2022CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilitiescited by 249
  4. 2023Fine-tuning Language Models for Factualitycited by 221
  5. 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 211
  6. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 206
  7. 2019Do NLP Models Know Numbers? Probing Numeracy in Embeddingscited by 159
  8. 2022Ask Me Anything: A simple strategy for prompting language modelscited by 155
  9. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  10. 2022PromptChainer: Chaining Large Language Model Prompts through Visual Programmingcited by 134
  11. 2020Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertaintycited by 129
  12. 2020The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Modelscited by 125
  13. 2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 123
  14. 2023Is ChatGPT a Highly Fluent Grammatical Error Correction System? A Comprehensive Evaluationcited by 97
  15. 2019explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learningcited by 70
  16. 2018Seq2Seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Modelscited by 69
  17. 2022Creative Writing with an AI-Powered Writing Assistant: Perspectives from Professional Writerscited by 69
  18. 2023Graphologue: Exploring Large Language Model Responses with Interactive Diagramscited by 66
  19. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  20. 2023Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Modelscited by 48
  21. 2022Beyond Text Generation: Supporting Writers with Continuous Automatic Text Summariescited by 47
  22. 2021It is AI's Turn to Ask Humans a Question: Question-Answer Pair Generation for Children's Story Bookscited by 23
  23. 2022Mapping the Design Space of Human-AI Interaction in Text Summarizationcited by 15
  24. 2022Visual Comparison of Language Model Adaptationcited by 14
  25. 2021VBridge: Connecting the Dots Between Features and Data to Explain Healthcare Modelscited by 12
  26. 2022Improving Wikipedia Verifiability with AIcited by 11
  27. 2021Backtranslation Feedback Improves User Confidence in MT, Not Qualitycited by 10
  28. 2022Beware the Rationalization Trap! When Language Model Explainability Diverges from our Mental Models of Languagecited by 9
  29. 2023Mapping the Design Space of Interactions in Human-AI Text Co-creation Taskscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 14; 2026: 6.

23 papers.

Sort:

By area:

  1. 2025Can Large Language Models Match Tutoring System Adaptivity? A Benchmarking Studycited by 18
  2. 2025Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Reviewcited by 16
  3. 2025Consistency in Language Models: Current Landscape, Challenges, and Future Directionscited by 14
  4. 2024One vs. Many: Comprehending Accurate Information from Multiple Erroneous and Inconsistent AI Generationscited by 13
  5. 2025SPHERE: An Evaluation Card for Human-AI Systemscited by 9
  6. 2025Semantic-guided Diverse Decoding for Large Language Modelcited by 9
  7. 2024Understanding Large Language Model Behaviors through Interactive Counterfactual Generation and Analysiscited by 8
  8. 2025Surfacing Variations to Calibrate Perceived Reliability of MLLM-generated Image Descriptionscited by 6
  9. 2025RAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generationcited by 4
  10. 2025Semantic Commit: Helping Users Update Intent Specifications for AI Memory at Scalecited by 3
  11. 2025DxHF: Providing High-Quality Human Feedback for LLM Alignment via Interactive Decompositioncited by 3
  12. 2025Are LLMs reliable? An exploration of the reliability of large language models in clinical note generationcited by 2
11 more
  1. 2024LEAF: Learning and Evaluation Augmented by Fact-Checking to Improve Factualness in Large Language Modelscited by 1
  2. 2025FactFlow: Automatic Fact Sheet Generation and Customization from Tabular Dataset via AI Chain Design & Implementationcited by 1
  3. 2025Hide or Highlight: Understanding the Impact of Factuality Expression on User Trustcited by 1
  4. 2026Learning from AVA: Early Lessons from a Curated and Trustworthy Generative AI for Policy and Development Researchcited by 1
  5. 2025Beyond Autocomplete: Designing CopilotLens Towards Transparent and Explainable AI Coding Agentscited by 0
  6. 2025VIDEE: Visual and Interactive Decomposition, Execution, and Evaluation of Text Analytics with Intelligent Agentscited by 0
  7. 2026ReBeCA: Unveiling Interpretable Behavior Hierarchy behind the Iterative Self-Reflection of Language Models with Causal Analysiscited by 0
  8. 2026Beyond One Output: Visualizing and Comparing Distributions of Language Model Generationscited by 0
  9. 2026Visual Fingerprints for LLM Generation Comparisoncited by 0
  10. 2026Task Abstention for Large Language Models in Code Generationcited by 0
  11. 2026Signals of AI Hallucination: Designing Hallucination-Aware Cues for Embodied Conversational Agents in VRcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Models
  2. 2024Internal Consistency and Self-Feedback in Large Language Models: A Survey
  3. 2024FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluation
  4. 2026How Large Language Models Balance Internal Knowledge with User and Document Assertions
  5. 2025Are the Hidden States Hiding Something? Testing the Limits of Factuality-Encoding Capabilities in LLMs
  6. 2024FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"
  7. 2024To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts
  8. 2024To Know or Not To Know? Analyzing Self-Consistency of Large Language Models under Ambiguity