mlsanityData to 14 Oct 2026

Pre-training Language Models for Comparative Reasoning

arXiv 2305.14457 ↗First posted 23 May 2023cs.CLcited by 6

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Comparative reasoning is a process of comparing objects, concepts, or entities to draw conclusions, which constitutes a fundamental cognitive ability. In this paper, we propose a novel framework to pre-train language models for enhancing their abilities of comparative reasoning over texts. While there have been approaches for NLP tasks that require comparative reasoning, they suffer from costly manual data labeling and limited generalizability to different tasks. Our approach introduces a novel method of collecting scalable data for text-based entity comparison, which leverages both structured and unstructured data. Moreover, we present a framework of pre-training language models via three novel objectives on comparative reasoning. Evaluation on downstream tasks including comparative question answering, question generation, and summarization shows that our pre-training framework significantly improves the comparative reasoning abilities of language models, especially under low-resource conditions. This work also releases the first integrated benchmark for comparative reasoning.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 27 of its 37 references that are in the map. First posted on arXiv 23 May 2023.

Builds on

27 of 37 references.

References · 27

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  3. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  4. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  5. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  6. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  7. 2017Reading Wikipedia to Answer Open-Domain Questionscited by 777
  8. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 573
  9. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 528
  10. 2019KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representationcited by 356
  11. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 319
  12. 2019K-BERT: Enabling Language Representation with Knowledge Graphcited by 304
15 more
  1. 2020DART: Open-Domain Structured Data Record to Text Generationcited by 145
  2. 2020Investigating Pretrained Language Models for Graph-to-Text Generationcited by 117
  3. 2020KG-BART: Knowledge Graph-Augmented BART for Generative Commonsense Reasoningcited by 106
  4. 2019Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Modelcited by 102
  5. 2020JAKET: Joint Pre-training of Knowledge Graph and Language Understandingcited by 73
  6. 2020ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learningcited by 72
  7. 2021JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphscited by 58
  8. 2022A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methodscited by 42
  9. 2020Unsupervised Multi-hop Question Answering by Question Generationcited by 38
  10. 2021Comparative Opinion Summarization via Collaborative Decodingcited by 18
  11. 2020Knowledge-Aware Procedural Text Understanding with Multi-Stage Trainingcited by 17
  12. 2022Empowering Language Models with Knowledge Graph Reasoning for Question Answeringcited by 14
  13. 2021Relation-Guided Pre-Training for Open-Domain Question Answeringcited by 7
  14. 2022A Unified Encoder-Decoder Framework with Entity Memorycited by 7
  15. 2018Comparative Document Summarisation via Classificationcited by 2

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 2.

6 papers.

  1. 2023Can Knowledge Graphs Reduce Hallucinations in LLMs? : A Surveycited by 92
  2. 2023Auto-Instruct: Automatic Instruction Generation and Ranking for Black-Box Language Modelscited by 21
  3. 2023User Modeling in the Era of Large Language Models: Current Research and Future Directionscited by 18
  4. 2023Predicting Text Preference Via Structured Comparative Reasoningcited by 6
  5. 2024Unveiling Comparative Sentiments in Vietnamese Product Reviews: A Sequential Classification Frameworkcited by 1
  6. 2024Can MLLMs generate human-like feedback in grading multimodal short answers?cited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2022ALERT: Adapting Language Models to Reasoning Tasks
  2. 2026Building Multilingual Bridges: Data Mixing as the Pillar of Generalization for In-Language Reasoning
  3. 2022APOLLO: A Simple Approach for Adaptive Pretraining of Language Models for Logical Reasoning
  4. 2024Conceptual and Unbiased Reasoning in Language Models
  5. 2025Evaluating the Meta- and Object-Level Reasoning of Large Language Models for Question Answering
  6. 2022Towards Reasoning in Large Language Models: A Survey
  7. 2025Advancing Reasoning in Large Language Models: Promising Methods and Approaches
  8. 2025ReasoningWeekly: A General Knowledge and Verbal Reasoning Challenge for Large Language Models