mlsanityData to 7 Oct 2026

Advancing Transformer's Capabilities in Commonsense Reasoning

arXiv 2310.06803 ↗First posted 10 Oct 2023cs.CL cs.AI cs.LGcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Recent advances in general purpose pre-trained language models have shown great potential in commonsense reasoning. However, current works still perform poorly on standard commonsense reasoning benchmarks including the Com2Sense Dataset. We argue that this is due to a disconnect with current cutting-edge machine learning methods. In this work, we aim to bridge the gap by introducing current ML-based methods to improve general purpose pre-trained language models in the task of commonsense reasoning. Specifically, we experiment with and systematically evaluate methods including knowledge transfer, model ensemble, and introducing an additional pairwise contrastive objective. Our best model outperforms the strongest previous works by ~15% absolute gains in Pairwise Accuracy and ~8.7% absolute gains in Standard Accuracy.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 10 of its 11 references that are in the map. First posted on arXiv 10 Oct 2023.

Builds on

10 of 11 references.

References · 10

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  6. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  7. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  8. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanationscited by 159
  9. 2020SemEval-2020 Task 4: Commonsense Validation and Explanationcited by 42
  10. 2021COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentencescited by 40

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022A Theoretically Grounded Benchmark for Evaluating Machine Commonsense
  2. 2022Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Survey
  3. 2023Commonsense Reasoning for Conversational AI: A Survey of the State of the Art
  4. 2025ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question Answering
  5. 2022CIKQA: Learning Commonsense Inference with a Unified Knowledge-in-the-loop QA Paradigm
  6. 2024From Data to Commonsense Reasoning: The Use of Large Language Models for Explainable AI
  7. 2025Com2: A Causal-Guided Benchmark for Exploring Complex Commonsense Reasoning in Large Language Models
  8. 2022Evaluate Confidence Instead of Perplexity for Zero-shot Commonsense Reasoning