mlsanityData to 7 Oct 2026

Establishing Trustworthiness: Rethinking Tasks and Model Evaluation

arXiv 2310.05442 ↗First posted 9 Oct 2023cs.CLcited by 4

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Language understanding is a multi-faceted cognitive capability, which the Natural Language Processing (NLP) community has striven to model computationally for decades. Traditionally, facets of linguistic intelligence have been compartmentalized into tasks with specialized model architectures and corresponding evaluation protocols. With the advent of large language models (LLMs) the community has witnessed a dramatic shift towards general purpose, task-agnostic approaches powered by generative models. As a consequence, the traditional compartmentalized notion of language tasks is breaking down, followed by an increasing challenge for evaluation and analysis. At the same time, LLMs are being deployed in more real-world scenarios, including previously unforeseen zero-shot setups, increasing the need for trustworthy and reliable systems. Therefore, we argue that it is time to rethink what constitutes tasks and model evaluation in NLP, and pursue a more holistic view on language, placing trustworthiness at the center. Towards this goal, we review existing compartmentalized approaches for understanding the origins of a model's functional capacity, and provide recommendations for more multi-faceted evaluation protocols.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 56 of its 79 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

56 of 79 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb
  2. 2023Augmented Language Models: a Survey15 Feb

Other references · 54

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  12. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
38 more
  1. 2022Emergent Abilities of Large Language Modelscited by 2,483
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  3. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  4. 2023HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Facecited by 1,173
  5. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  6. 2022Editing Models with Task Arithmeticcited by 1,076
  7. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  8. 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 864
  9. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  10. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  11. 2021Editing Factual Knowledge in Language Modelscited by 548
  12. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  13. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  14. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  15. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 343
  16. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 340
  17. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  18. 2017Distilling a Neural Network Into a Soft Decision Treecited by 230
  19. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  20. 2020Probing Pretrained Language Models for Lexical Semanticscited by 156
  21. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  22. 2020Making deep neural networks right for the right scientific reasons by interacting with their explanationscited by 141
  23. 2022On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?cited by 137
  24. 2023Faithfulness Tests for Natural Language Explanationscited by 107
  25. 2023Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Taskscited by 102
  26. 2022Finding Skill Neurons in Pre-trained Transformer-based Language Modelscited by 65
  27. 2020Targeting the Benchmark: On Methodology in Current Natural Language Processing Researchcited by 44
  28. 2023"According to ...": Prompting Language Models Improves Quoting from Pre-Training Datacited by 42
  29. 2021The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Failcited by 38
  30. 2017Limits of End-to-End Learningcited by 35
  31. 2023A Systematic Literature Review of User Trust in AI-Enabled Systems: An HCI Perspectivecited by 33
  32. 2023The ROOTS Search Tool: Data Transparency for LLMscited by 30
  33. 2023The Dangers of trusting Stochastic Parrots: Faithfulness and Trust in Open-domain Conversational Question Answeringcited by 26
  34. 2020Linguistic Profiling of a Neural Language Modelcited by 23
  35. 2023What's the Meaning of Superhuman Performance in Today's NLU?cited by 22
  36. 2022Informativeness and Invariance: Two Perspectives on Spurious Correlations in Natural Languagecited by 21
  37. 2022Neural Networks are Decision Treescited by 16
  38. 2023How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuningcited by 14

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025DaKultur: Evaluating the Cultural Awareness of Language Models for Danish with Native Speakerscited by 3
  2. 2025From Noise to Signal to Selbstzweck: Reframing Human Label Variation in the Era of Post-training in NLPcited by 3
  3. 2026Controlling Distributional Bias in Multi-Round LLM Generation via KL-Optimized Fine-Tuningcited by 1
  4. 2026VeriGraph: Towards Verifiable Data-Analytic Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From Interpretability to Control: Insights from Six Years of the TrustNLP Workshop
  2. 2025Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanations
  3. 2023Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks
  4. 2026Comprehensive Evaluation of Large Language Model Responses: A Multi-Factor Scoring System
  5. 2024LLMs' Understanding of Natural Language Revealed
  6. 2024Can Large Language Models be Trusted for Evaluation? Scalable Meta-Evaluation of LLMs as Evaluators via Agent Debate
  7. 2022Curriculum: A Broad-Coverage Benchmark for Linguistic Phenomena in Natural Language Understanding
  8. 2026The ACUTE Protocol: Operationalizing Language Model Activations for Better Calibration, Utility, and Trust