mlsanityData to 13 Oct 2026

Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoning

arXiv 2107.02794 ↗First posted 6 Jul 2021cs.AI cs.CL cs.LGcited by 85

Not on the map. Why?

Cited by 37 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Human reasoning can often be understood as an interplay between two systems: the intuitive and associative ("System 1") and the deliberative and logical ("System 2"). Neural sequence models -- which have been increasingly successful at performing complex, structured tasks -- exhibit the advantages and failure modes of System 1: they are fast and learn patterns from data, but are often inconsistent and incoherent. In this work, we seek a lightweight, training-free means of improving existing System 1-like sequence models by adding System 2-inspired logical reasoning. We explore several variations on this theme in which candidate generations from a neural sequence model are examined for logical consistency by a symbolic reasoning module, which can either accept or reject the generations. Our approach uses neural inference to mediate between the neural System 1 and the logical System 2. Results in robust story generation and grounded instruction-following show that this approach can increase the coherence and accuracy of neurally-based generations.

Key facts. Cited by 85 papers in the map (data through 13 Oct 2026). Builds on 42 of its 59 references that are in the map. First posted on arXiv 6 Jul 2021.

Builds on

42 of 59 references.

References · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,884
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,284
  5. 2019The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervisioncited by 371
  6. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 310
  7. 2020Inductive Biases for Deep Learning of Higher-Level Cognitioncited by 273
  8. 2020Neurosymbolic AI: The 3rd Wavecited by 254
  9. 2021Implicit Representations of Meaning in Neural Language Modelscited by 179
  10. 2020Recipes for Safety in Open-domain Chatbotscited by 170
  11. 2021Constrained Language Models Yield Few-Shot Semantic Parserscited by 163
  12. 2020DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learningcited by 161
30 more
  1. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 158
  2. 2020NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraintscited by 146
  3. 2018Dialogue Natural Language Inferencecited by 135
  4. 2017RobustFill: Neural Program Learning under Noisy I/Ocited by 125
  5. 2020Task-Oriented Dialogue as Dataflow Synthesiscited by 122
  6. 2018Learning to Write with Cooperative Discriminatorscited by 119
  7. 2020A Benchmark for Systematic Generalization in Grounded Language Understandingcited by 108
  8. 2016Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervisioncited by 102
  9. 2020Residual Energy-Based Models for Text Generationcited by 95
  10. 2018CGMH: Constrained Sentence Generation by Metropolis-Hastings Samplingcited by 93
  11. 2020Span-based Semantic Parsing for Compositional Generalizationcited by 77
  12. 2021Neural Production Systems: Learning Rule-Governed Visual Dynamicscited by 75
  13. 2020Learning Compositional Rules via Neural Program Synthesiscited by 74
  14. 2020I like fish, especially dolphins: Addressing Contradictions in Dialogue Modelingcited by 68
  15. 2017Event Representations for Automated Story Generation with Deep Neural Netscited by 62
  16. 2020Word meaning in minds and machinescited by 51
  17. 2020Neural-Symbolic Integration: A Compositional Perspectivecited by 45
  18. 2018A Skeleton-Based Model for Promoting Coherence Among Sentences in Narrative Story Generationcited by 42
  19. 2020Controlling Style in Generated Dialoguecited by 38
  20. 2020PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long Text Generationcited by 35
  21. 2019Pragmatically Informative Text Generationcited by 32
  22. 2020AutoQA: From Databases To QA Semantic Parsers With Only Synthetic Training Datacited by 29
  23. 2018Learning to Reason with Third-Order Tensor Productscited by 27
  24. 2020Generative Language-Grounded Policy in Vision-and-Language Navigation with Bayes' Rulecited by 22
  25. 2016Learning Executable Semantic Parsers for Natural Language Understandingcited by 21
  26. 2017Cross-domain Semantic Parsing via Paraphrasingcited by 19
  27. 2020Unnatural Language Processing: Bridging the Gap Between Synthetic and Natural Language Datacited by 16
  28. 2020Think before you act: A simple baseline for compositional generalizationcited by 11
  29. 2016Coupling Distributed and Symbolic Execution for Natural Language Queriescited by 9
  30. 2018Value-based Search in Execution Space for Mapping Instructions to Programscited by 2

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 2; 2022: 24; 2023: 26; 2024: 24; 2025: 5; 2026: 4.

85 papers.

Sort:

By area:

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,657
  2. 2022PAL: Program-aided Language Modelscited by 647
  3. 2022Using cognitive psychology to understand GPT-3cited by 395
  4. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 332
  5. 2022Language models show human-like content effects on reasoning taskscited by 172
  6. 2023SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Taskscited by 166
  7. 2022Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanationscited by 159
  8. 2022Rethinking with Retrieval: Faithful Large Language Model Inferencecited by 145
  9. 2022Thinking Fast and Slow in Large Language Modelscited by 136
  10. 2022Who is GPT-3? An Exploration of Personality, Values and Demographicscited by 112
  11. 2023A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writingcited by 98
  12. 2023Minding Language Models' (Lack of) Theory of Mind: A Plug-and-Play Multi-Character Belief Trackercited by 95
38 more
  1. 2023Think Twice: Perspective-Taking Improves Large Language Models' Theory-of-Mind Capabilitiescited by 86
  2. 2022Solving Math Word Problems via Cooperative Reasoning induced Language Modelscited by 77
  3. 2024Calibrating Large Language Models with Sample Consistencycited by 73
  4. 2023PEARL: Prompting Large Language Models to Plan and Execute Actions Over Long Documentscited by 61
  5. 2023Harnessing the Power of Large Language Models for Natural Language to First-Order Logic Translationcited by 59
  6. 2023Coupling Large Language Models with Logic Programming for Robust and General Reasoning from Textcited by 59
  7. 2022Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning taskscited by 56
  8. 2024Visual Agents as Fast and Slow Thinkerscited by 53
  9. 2023SGP-TOD: Building Task Bots Effortlessly via Schema-Guided LLM Promptingcited by 39
  10. 2024System-1.x: Learning to Balance Fast and Slow Planning with Language Modelscited by 35
  11. 2023Enhancing Zero-Shot Chain-of-Thought Reasoning in Large Language Models through Logiccited by 34
  12. 2023Leveraging Large Language Models to Generate Answer Set Programscited by 33
  13. 2022Neuro-Symbolic Procedural Planning with Commonsense Promptingcited by 28
  14. 2022Faithfulness in Natural Language Generation: A Systematic Survey of Analysis, Evaluation and Optimization Methodscited by 27
  15. 2022Relational Memory Augmented Language Modelscited by 24
  16. 2023Learning to Solve Constraint Satisfaction Problems with Recurrent Transformercited by 21
  17. 2023Improved Logical Reasoning of Language Models via Differentiable Symbolic Programmingcited by 18
  18. 2024Arithmetic Reasoning with LLM: Prolog Generation & Permutationcited by 17
  19. 2023BatchPrompt: Accomplish more with lesscited by 16
  20. 2023Symbol-LLM: Leverage Language Models for Symbolic System in Visual Human Activity Reasoningcited by 16
  21. 2024Reliable Reasoning Beyond Natural Languagecited by 14
  22. 2022Improving Systematic Generalization Through Modularity and Augmentationcited by 13
  23. 2022Do language models have coherent mental models of everyday things?cited by 13
  24. 2023Explicit Planning Helps Language Models in Logical Reasoningcited by 12
  25. 2024Learning to Plan for Language Modeling from Unlabeled Datacited by 12
  26. 2023SimOAP: Improve Coherence and Consistency in Persona-based Dialogue Generation via Over-sampling and Post-evaluationcited by 11
  27. 2024The Role of Foundation Models in Neuro-Symbolic Learning and Reasoningcited by 11
  28. 2022AnyTOD: A Programmable Task-Oriented Dialog Systemcited by 10
  29. 2024Strategies for Improving NL-to-FOL Translation with LLMs: Data Generation, Incremental Fine-Tuning, and Verificationcited by 10
  30. 2024Neuro-Symbolic AI: Explainability, Challenges, and Future Trendscited by 10
  31. 2025LLM+AL: Bridging Large Language Models and Action Languages for Complex Reasoning about Actionscited by 10
  32. 2024Efficient Rectification of Neuro-Symbolic Reasoning Inconsistencies by Abductive Reflectioncited by 9
  33. 2022Neuro-symbolic Explainable Artificial Intelligence Twin for Zero-touch IoE in Wireless Networkcited by 8
  34. 2024Towards Building Specialized Generalist AI with System 1 and System 2 Fusioncited by 8
  35. 2024LLASP: Fine-tuning Large Language Models for Answer Set Programmingcited by 8
  36. 2022CoRRPUS: Code-based Structured Prompting for Neurosymbolic Story Understandingcited by 7
  37. 2024Towards Automatic Composition of ASP Programs from Natural Language Specificationscited by 7
  38. 2024Bayesian Statistical Modeling with Predictors from LLMscited by 7

50 most cited of 85 All 85 (JSON)