mlsanityData to 14 Oct 2026

From Test-Taking to Test-Making: Examining LLM Authoring of Commonsense Assessment Items

arXiv 2410.14897 ↗First posted 18 Oct 2024cs.CL cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

LLMs can now perform a variety of complex writing tasks. They also excel in answering questions pertaining to natural language inference and commonsense reasoning. Composing these questions is itself a skilled writing task, so in this paper we consider LLMs as authors of commonsense assessment items. We prompt LLMs to generate items in the style of a prominent benchmark for commonsense reasoning, the Choice of Plausible Alternatives (COPA). We examine the outcome according to analyses facilitated by the LLMs and human annotation. We find that LLMs that succeed in answering the original COPA benchmark are also more successful in authoring their own items.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 15 of its 31 references that are in the map. First posted on arXiv 18 Oct 2024.

Builds on

15 of 31 references.

References · 15

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2020Scaling Laws for Neural Language Modelscited by 6,586
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  4. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  5. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  6. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  7. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 734
  8. 2023The Falcon Series of Open Language Modelscited by 535
  9. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  10. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  11. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  12. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
3 more
  1. 2024A Note on Shumailov et al. (2024): `AI Models Collapse When Trained on Recursively Generated Data'cited by 115
  2. 2021Generate, Annotate, and Learn: NLP with Synthetic Textcited by 42
  3. 2024GPTs Are Multilingual Annotators for Sequence Generation Taskscited by 11

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Enhancing LLM Robustness to Perturbed Instructions: An Empirical Studycited by 17
  2. 2025What Has Been Lost with Synthetic Evaluation?cited by 9

Similar, not cited

Close in wording, no citation either way

  1. 2025Towards Quantifying Commonsense Reasoning with Mechanistic Insights
  2. 2024From Data to Commonsense Reasoning: The Use of Large Language Models for Explainable AI
  3. 2023Are LLMs the Master of All Trades? : Exploring Domain-Agnostic Reasoning Skills of LLMs
  4. 2026mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages?
  5. 2024A Survey of Useful LLM Evaluation
  6. 2025Analysis of instruction-based LLMs' capabilities to score and judge text-input problems in an academic setting
  7. 2024Enhancing LLM Evaluations: The Garbling Trick
  8. 2025What Kind of Reasoning (if any) is an LLM actually doing? On the Stochastic Nature and Abductive Appearance of Large Language Models