mlsanityData to 14 Oct 2026

CogniBench: A Legal-inspired Framework and Dataset for Assessing Cognitive Faithfulness of Large Language Models

arXiv 2505.20767 ↗First posted 27 May 2025cs.CL cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Faithfulness hallucinations are claims generated by a Large Language Model (LLM) not supported by contexts provided to the LLM. Lacking assessment standards, existing benchmarks focus on "factual statements" that rephrase source materials while overlooking "cognitive statements" that involve making inferences from the given context. Consequently, evaluating and detecting the hallucination of cognitive statements remains challenging. Inspired by how evidence is assessed in the legal domain, we design a rigorous framework to assess different levels of faithfulness of cognitive statements and introduce the CogniBench dataset where we reveal insightful statistics. To keep pace with rapidly evolving LLMs, we further develop an automatic annotation pipeline that scales easily across different models. This results in a large-scale CogniBench-L dataset, which facilitates training accurate detectors for both factual and cognitive hallucinations. We release our model and datasets at: https://github.com/FUTUREEEEEE/CogniBench

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 22 of its 40 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

22 of 40 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  11. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  12. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,292
10 more
  1. 2022A Survey on In-context Learningcited by 721
  2. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  3. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 283
  4. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 265
  5. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 244
  6. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 236
  7. 2024Fine-grained Hallucination Detection and Editing for Language Modelscited by 149
  8. 2024Large Language Model Agent in Financial Trading: A Surveycited by 60
  9. 2024Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarkscited by 32
  10. 2024HalluDial: A Large-Scale Benchmark for Automatic Dialogue-Level Hallucination Evaluationcited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Disentangling Deception and Hallucination Failures in LLMscited by 0