mlsanityData to 14 Oct 2026

Who's the Best Detective? LLMs vs. MLs in Detecting Incoherent Fourth Grade Math Answers

arXiv 2304.11257 ↗First posted 21 Apr 2023cs.CL cs.AI cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Written answers to open-ended questions can have a higher long-term effect on learning than multiple-choice questions. However, it is critical that teachers immediately review the answers, and ask to redo those that are incoherent. This can be a difficult task and can be time-consuming for teachers. A possible solution is to automate the detection of incoherent answers. One option is to automate the review with Large Language Models (LLM). In this paper, we analyze the responses of fourth graders in mathematics using three LLMs: GPT-3, BLOOM, and YOU. We used them with zero, one, two, three and four shots. We compared their performance with the results of various classifiers trained with Machine Learning (ML). We found that LLMs perform worse than MLs in detecting incoherent answers. The difficulty seems to reside in recursive questions that contain both questions and answers, and in responses from students with typical fourth-grader misspellings. Upon closer examination, we have found that the ChatGPT model faces the same challenges.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 24 of its 57 references that are in the map. First posted on arXiv 21 Apr 2023.

Builds on

24 of 57 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  8. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  9. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  10. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  11. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 807
  12. 2022Towards Reasoning in Large Language Models: A Surveycited by 596
12 more
  1. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  2. 2023Augmented Language Models: a Surveycited by 403
  3. 2022Using cognitive psychology to understand GPT-3cited by 398
  4. 2018Transforming Question Answering Datasets Into Natural Language Inference Datasetscited by 99
  5. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 87
  6. 2023ChatGPT and Software Testing Education: Promises & Perilscited by 42
  7. 2020ANLIzing the Adversarial Natural Language Inference Datasetcited by 36
  8. 2018Solving Bongard Problems with a Visual Language and Pragmatic Reasoningcited by 21
  9. 2023Creating a Large Language Model of a Philosophercited by 21
  10. 2022Survey on Automated Short Answer Grading with Deep Learning: from Word Embeddings to Transformerscited by 20
  11. 2020Comparative Evaluation of Pretrained Transfer Learning Models on Automatic Short Answer Gradingcited by 5
  12. 2022Towards a Solution to Bongard Problems: A Causal Approachcited by 4

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 2.

3 papers.

  1. 2024Can LLMs Master Math? Investigating Large Language Models on Math Stack Exchangecited by 22
  2. 2024OpenAI-o1 AB Testing: Does the o1 model really do good reasoning in math problem solving?cited by 4
  3. 2023Deep Natural Language Feature Learning for Interpretable Predictioncited by 1