mlsanityData to 14 Oct 2026

IDOL: Indicator-oriented Logic Pre-training for Logical Reasoning

arXiv 2306.15273 ↗First posted 27 Jun 2023cs.CL cs.AIcited by 3

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

In the field of machine reading comprehension (MRC), existing systems have surpassed the average performance of human beings in many tasks like SQuAD. However, there is still a long way to go when it comes to logical reasoning. Although some methods for it have been put forward, they either are designed in a quite complicated way or rely too much on external structures. In this paper, we proposed IDOL (InDicator-Oriented Logic Pre-training), an easy-to-understand but highly effective further pre-training task which logically strengthens the pre-trained models with the help of 6 types of logical indicators and a logically rich dataset LGP (LoGic Pre-training). IDOL achieves state-of-the-art performance on ReClor and LogiQA, the two most representative benchmarks in logical reasoning MRC, and is proven to be capable of generalizing to different pre-trained models and other types of MRC benchmarks like RACE and SQuAD 2.0 while keeping competitive general language understanding ability through testing on tasks in GLUE. Besides, at the beginning of the era of large language models, we take several of them like ChatGPT into comparison and find that IDOL still shows its advantage.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 24 of its 33 references that are in the map. First posted on arXiv 27 Jun 2023.

Builds on

24 of 33 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022LogiGAN: Learning Logical Reasoning via Adversarial Pre-training18 May
  2. 2022MERIt: Meta-Path Guided Contrastive Learning for Logical Reasoning1 Mar

Other references · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  8. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  9. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  10. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  11. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  12. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 843
10 more
  1. 2020Revisiting Pre-Trained Models for Chinese Natural Language Processingcited by 266
  2. 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 217
  3. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  4. 2018Transforming Question Answering Datasets Into Natural Language Inference Datasetscited by 99
  5. 2020ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learningcited by 72
  6. 2021Logic-Driven Context Extension and Data Augmentation for Logical Reasoning of Textcited by 46
  7. 2021DAGN: Discourse-Aware Graph Network for Logical Reasoningcited by 28
  8. 2022PERT: Pre-training BERT with Permuted Language Modelcited by 23
  9. 2022AdaLoGN: Adaptive Logic Graph Network for Reasoning-Based Machine Reading Comprehensioncited by 8
  10. 2022Logiformer: A Two-Branch Graph Transformer Network for Interpretable Logical Reasoningcited by 8

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 1.

3 papers.

  1. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  2. 2023CLOMO: Counterfactual Logical Modification with Large Language Modelscited by 7
  3. 2025LFC-DA: Logical Formula-Controlled Data Augmentation for Enhanced Logical Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning
  2. 2022APOLLO: A Simple Approach for Adaptive Pretraining of Language Models for Logical Reasoning
  3. 2023Boosting Logical Reasoning in Large Language Models through a New Framework: The Graph of Thought
  4. 2024LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models
  5. 2025Human-Level Reasoning: A Comparative Study of Large Language Models on Logical and Abstract Reasoning
  6. 2025MMReason: An Open-Ended Multi-Modal Multi-Step Reasoning Benchmark for MLLMs Toward AGI
  7. 2025DivLogicEval: A Framework for Benchmarking Logical Reasoning Evaluation in Large Language Models
  8. 2025MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMs