mlsanityData to 7 Oct 2026

From LSAT: The Progress and Challenges of Complex Reasoning

arXiv 2108.00648 ↗First posted 2 Aug 2021cs.CLcited by 43

Not on the map. Why?

Cited by 24 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Complex reasoning aims to draw a correct inference based on complex rules. As a hallmark of human intelligence, it involves a degree of explicit reading comprehension, interpretation of logical knowledge and complex rule application. In this paper, we take a step forward in complex reasoning by systematically studying the three challenging and domain-general tasks of the Law School Admission Test (LSAT), including analytical reasoning, logical reasoning and reading comprehension. We propose a hybrid reasoning system to integrate these three tasks and achieve impressive overall performance on the LSAT tests. The experimental results demonstrate that our system endows itself a certain complex reasoning ability, especially the fundamental reading comprehension and challenging logical reasoning capacities. Further analysis also shows the effectiveness of combining the pre-trained models with the task-specific reasoning module, and integrating symbolic knowledge into discrete interpretable reasoning steps in complex reasoning. We further shed a light on the potential future directions, like unsupervised symbolic knowledge extraction, model interpretability, few-shot learning and comprehensive benchmark for complex reasoning.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 61 of its 113 references that are in the map. First posted on arXiv 2 Aug 2021.

Builds on

61 of 113 references.

References · 61

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  6. 2017Graph Attention Networkscited by 8,854
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2020Longformer: The Long-Document Transformercited by 3,362
  10. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,837
38 more
  1. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  2. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,065
  3. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  4. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  5. 2018CoQA: A Conversational Question Answering Challengecited by 674
  6. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  7. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 642
  8. 2018QuAC : Question Answering in Contextcited by 379
  9. 2015Neural Module Networkscited by 375
  10. 2019Abductive Commonsense Reasoningcited by 300
  11. 2019Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoningcited by 281
  12. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  13. 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 211
  14. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 198
  15. 2017Neural-Symbolic Learning and Reasoning: A Survey and Interpretationcited by 195
  16. 2017Constructing Datasets for Multi-hop Reading Comprehension Across Documentscited by 190
  17. 2019Neural-Symbolic Computing: An Effective Methodology for Principled Integration of Machine Learning and Reasoningcited by 149
  18. 2019"Going on a vacation" takes longer than "Going for a walk": A Study of Temporal Commonsense Understandingcited by 121
  19. 2016Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervisioncited by 101
  20. 2017Logic Tensor Networks for Semantic Image Interpretationcited by 92
  21. 2019Counterfactual Story Reasoning and Generationcited by 87
  22. 2017Semi-supervised sequence tagging with bidirectional language modelscited by 86
  23. 2020Retrospective Reader for Machine Reading Comprehensioncited by 86
  24. 2018Leveraging Grammar and Reinforcement Learning for Neural Program Synthesiscited by 83
  25. 2018Event2Mind: Commonsense Inference on Events, Intents, and Reactionscited by 82
  26. 2018One-Shot Relational Learning for Knowledge Graphscited by 78
  27. 2019Neural Module Networks for Reasoning over Textcited by 65
  28. 2018Translating a Math Word Problem to an Expression Treecited by 57
  29. 2021AR-LSAT: Investigating Analytical Reasoning of Textcited by 49
  30. 2021Logic-Driven Context Extension and Data Augmentation for Logical Reasoning of Textcited by 46
  31. 2019Multi-hop Question Answering via Reasoning Chainscited by 42
  32. 2019R4C: A Benchmark for Evaluating RC Systems to Get the Right Answer for the Right Reasoncited by 36
  33. 2018Extending a Parser to Distant Domains Using a Few Dozen Partially Annotated Examplescited by 34
  34. 2020Conversational Neuro-Symbolic Commonsense Reasoningcited by 28
  35. 2021DAGN: Discourse-Aware Graph Network for Logical Reasoningcited by 28
  36. 2020LogicalFactChecker: Leveraging Logical Operations for Fact Checking with Graph Module Networkcited by 26
  37. 2020Learning by Fixing: Solving Math Word Problems with Weak Supervisioncited by 25
  38. 2021Neural Multi-Hop Reasoning With Logical Rules on Biomedical Knowledge Graphscited by 25

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 5; 2024: 12; 2025: 13; 2026: 8.

43 papers.

Sort:

By area:

  1. 2025s1: Simple test-time scalingcited by 1,311
  2. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  3. 2022Reasoning with Language Model Prompting: A Surveycited by 251
  4. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4cited by 202
  5. 2024The Effect of Sampling Temperature on Problem Solving in Large Language Modelscited by 168
  6. 2024Language models scale reliably with over-training and on downstream taskscited by 105
  7. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 94
  8. 2024A Review of Prominent Paradigms for LLM-Based Agents: Tool Use (Including RAG), Planning, and Feedback Learningcited by 74
  9. 2025Learn to Reason Efficiently with Adaptive Length-based Reward Shapingcited by 74
  10. 2024The Benefits of a Concise Chain of Thought on Problem-Solving in Large Language Modelscited by 62
  11. 2025Reinforcement Learning Finetunes Small Subnetworks in Large Language Modelscited by 46
  12. 2026Parcae: Scaling Laws For Stable Looped Language Modelscited by 45
31 more
  1. 2024Self-Reflection in LLM Agents: Effects on Problem-Solving Performancecited by 38
  2. 2025Inference-Time Computations for LLM Reasoning and Planning: A Benchmark and Insightscited by 30
  3. 2022E-KAR: A Benchmark for Rationalizing Natural Language Analogical Reasoningcited by 28
  4. 2024Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMscited by 28
  5. 2024When LLMs Meet Cunning Texts: A Fallacy Understanding Benchmark for Large Language Modelscited by 22
  6. 2025A Survey on LLM Test-Time Compute via Search: Tasks, LLM Profiling, Search Algorithms, and Relevant Frameworkscited by 21
  7. 2024Symbolic Working Memory Enhances Language Models for Complex Rule Applicationcited by 17
  8. 2023GLoRE: Evaluating Logical Reasoning of Large Language Modelscited by 13
  9. 2025LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviewscited by 13
  10. 2022LogiGAN: Learning Logical Reasoning via Adversarial Pre-trainingcited by 11
  11. 2023The Confidence-Competence Gap in Large Language Models: A Cognitive Studycited by 9
  12. 2025Comparative Analysis Based on DeepSeek, ChatGPT, and Google Gemini: Features, Techniques, Performance, Future Prospectscited by 9
  13. 2022AdaLoGN: Adaptive Logic Graph Network for Reasoning-Based Machine Reading Comprehensioncited by 8
  14. 2025Generative Evaluation of Complex Reasoning in Large Language Modelscited by 6
  15. 2023Unifying Structure Reasoning and Language Model Pre-training for Complex Reasoningcited by 4
  16. 2024A Study on Training and Developing Large Language Models for Behavior Tree Generationcited by 4
  17. 2022Disentangling Reasoning Capabilities from Language Models with Compositional Reasoning Transformerscited by 3
  18. 2024Logic Agent: Enhancing Validity with Logic Rule Invocationcited by 3
  19. 2025Nature's Insight: A Novel Framework and Comprehensive Analysis of Agentic Reasoning Through the Lens of Neurosciencecited by 3
  20. 2025Reinforcement Learning Fine-Tunes a Sparse Subnetwork in Large Language Modelscited by 2
  21. 2025LogiNumSynth: Synthesizing Joint Logical-Numerical Reasoning Problems for Language Modelscited by 2
  22. 2026Prefix Sliding for efficient test-time scalingcited by 2
  23. 2024Recurrent Alignment with Hard Attention for Hierarchical Text Ratingcited by 1
  24. 2026From Chains to Graphs: Self-Structured Reasoning for General-Domain LLMscited by 1
  25. 2026A dataset of rated conceptual argumentscited by 1
  26. 2024GRAMMAR: Grounded and Modular Methodology for Assessment of Closed-Domain Retrieval-Augmented Language Modelcited by 0
  27. 2025FSLI: An Interpretable Formal Semantic System for One-Dimensional Ordering Inferencecited by 0
  28. 2026Continually self-improving AIcited by 0
  29. 2026Toward Cross-Lingual Quality Classifiers for Multilingual Pretraining Data Selectioncited by 0
  30. 2026Nice Fold or Hero Call: Learning Budget-Efficient Thinking for Adaptive Reasoningcited by 0
  31. 2026From Process Loss to Assembly Bonus: Human-Grounded Diagnosis of Multi-Agent LLM Collaborationcited by 0