mlsanityData to 7 Oct 2026

A Simple Method for Commonsense Reasoning

arXiv 1806.02847 ↗First posted 7 Jun 2018cs.AI cs.CL cs.LGcited by 223

Not on the map. Why?

Abstract (arXiv)

Commonsense reasoning is a long-standing challenge for deep learning. For example, it is difficult to use neural networks to tackle the Winograd Schema dataset (Levesque et al., 2011). In this paper, we present a simple method for commonsense reasoning with neural networks, using unsupervised learning. Key to our method is the use of language models, trained on a massive amount of unlabled data, to score multiple choice questions posed by commonsense reasoning tests. On both Pronoun Disambiguation and Winograd Schema challenges, our models outperform previous state-of-the-art methods by a large margin, without using expensive annotated knowledge bases or hand-engineered features. We train an array of large RNN language models that operate at word or character level on LM-1-Billion, CommonCrawl, SQuAD, Gutenberg Books, and a customized corpus for this task and show that diversity of training data plays an important role in test performance. Further analysis also shows that our system successfully discovers important features of the context that decide the correct answer, indicating a good grasp of commonsense knowledge.

Key facts. Cited by 223 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 57; 2022: 56; 2023: 57; 2024: 35; 2025: 10; 2026: 8.

223 papers.

Sort:

By area:

  1. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  2. 2023A Survey of Large Language Modelscited by 2,537
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  5. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  6. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  7. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  8. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  9. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  10. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  11. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  12. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
38 more
  1. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  2. 2023A Comprehensive Overview of Large Language Modelscited by 639
  3. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  4. 2023The Falcon Series of Open Language Modelscited by 526
  5. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  6. 2021Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpuscited by 481
  7. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  8. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  9. 20218-bit Optimizers via Block-wise Quantizationcited by 395
  10. 2021Few-shot Learning with Multilingual Language Modelscited by 278
  11. 2022Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofscited by 278
  12. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  13. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  14. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  15. 2021Generated Knowledge Prompting for Commonsense Reasoningcited by 213
  16. 2021How Many Data Points is a Prompt Worth?cited by 197
  17. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  18. 2021Efficient Large Scale Language Modeling with Mixtures of Expertscited by 175
  19. 2023CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languagescited by 172
  20. 2022Language models show human-like content effects on reasoning taskscited by 171
  21. 2023MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformerscited by 164
  22. 2022Towards a Cleaner Document-Oriented Multilingual Crawled Corpuscited by 143
  23. 2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
  24. 2023The All-Seeing Project: Towards Panoptic Visual Recognition and Understanding of the Open Worldcited by 114
  25. 2021BERT: A Review of Applications in Natural Language Processing and Understandingcited by 109
  26. 2023Copyright Violations and Large Language Modelscited by 108
  27. 2024LoRA vs Full Fine-tuning: An Illusion of Equivalencecited by 102
  28. 2021UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmarkcited by 98
  29. 2021Luna: Linear Unified Nested Attentioncited by 96
  30. 2022Less is More: Task-aware Layer-wise Distillation for Language Model Compressioncited by 91
  31. 2022Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Modelscited by 90
  32. 2021ClimateBert: A Pretrained Language Model for Climate-Related Textcited by 89
  33. 2024Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Modelscited by 83
  34. 2023FP8-LM: Training FP8 Large Language Modelscited by 75
  35. 2021Probing Across Time: What Does RoBERTa Know and When?cited by 73
  36. 2023Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Promptingcited by 71
  37. 2021It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoningcited by 69
  38. 2023Prompting for Multimodal Hateful Meme Classificationcited by 68

50 most cited of 223 All 223 (JSON)