mlsanityData to 7 Oct 2026

GPT-Neo for commonsense reasoning -- a theoretical and practical lens

arXiv 2211.15593 ↗First posted 28 Nov 2022cs.CL cs.LGcited by 4

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Recent work has demonstrated substantial gains in pre-training large-language models (LLMs) followed by supervised fine-tuning on the downstream task. In this paper, we evaluate the performance of the GPT-neo model using 6 commonsense reasoning benchmark tasks. We aim to examine the performance of smaller models using the GPT-neo models against several larger model baselines such as GPT-3, Llama-2, MPT and Falcon. Upon fine-tuning with the appropriate set of hyperparameters, our model achieves competitive accuracy on several tasks. We also investigate and substantiate our results using attention-head visualization to better understand the model performance. Finally, we conduct various robustness tests using various methods to gauge the model performance under numerous settings.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 37 of its 57 references that are in the map. First posted on arXiv 28 Nov 2022.

Builds on

37 of 57 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Language Models of Code are Few-Shot Commonsense Learners13 Oct
  2. 2022Mind's Eye: Grounded Language Model Reasoning through Simulation11 Oct

Other references · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  12. 2020Longformer: The Long-Document Transformercited by 3,362
23 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  4. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  5. 2022In-context Learning and Induction Headscited by 930
  6. 2022A Generalist Agentcited by 802
  7. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  8. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  9. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  10. 2019Axial Attention in Multidimensional Transformerscited by 350
  11. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  12. 2020An Investigation of Why Overparameterization Exacerbates Spurious Correlationscited by 310
  13. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  14. 2016Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Reviewcited by 155
  15. 2022Ask Me Anything: A simple strategy for prompting language modelscited by 155
  16. 2021A Universal Law of Robustness via Isoperimetrycited by 150
  17. 2022The Debate Over Understanding in AI's Large Language Modelscited by 142
  18. 2022CommonsenseQA 2.0: Exposing the Limits of AI through Gamificationcited by 108
  19. 2019Evaluating Commonsense in Pre-trained Language Modelscited by 95
  20. 2022Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Datacited by 74
  21. 2021What to Pre-Train on? Efficient Intermediate Task Selectioncited by 62
  22. 2021A Systematic Investigation of Commonsense Knowledge in Large Language Modelscited by 56
  23. 2022Neural networks trained with SGD learn distributions of increasing complexitycited by 50

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 2.

4 papers.

  1. 2024Who Wrote This? The Key to Zero-Shot LLM-Generated Text Detection Is GECScorecited by 27
  2. 2025Revealing the Power of Post-Training for Small Language Models via Knowledge Distillationcited by 6
  3. 2025Frame In, Frame Out: Measuring Framing Bias in LLM-Generated News Summariescited by 2
  4. 2023Opinion Mining Using Population-tuned Generative Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Large Language Models Are Also Good Prototypical Commonsense Reasoners
  2. 2023Advancing Transformer's Capabilities in Commonsense Reasoning
  3. 2024Benchmarking Large Language Models for Persian: A Preliminary Study Focusing on ChatGPT
  4. 2024CARL-GT: Evaluating Causal Reasoning Capabilities of Large Language Models
  5. 2024From Data to Commonsense Reasoning: The Use of Large Language Models for Explainable AI
  6. 2026General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks
  7. 2025Benchmarking Reasoning Robustness in Large Language Models
  8. 2025mSCoRe: a Multilingual and Scalable Benchmark for Skill-based Commonsense Reasoning