# Symbolic Working Memory Enhances Language Models for Complex Rule Application

arXiv: https://arxiv.org/abs/2408.13654 · HTML: https://mlsanity.com/paper/2408.13654/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 26 of its 41 references that are in the map.
- First posted on arXiv 24 Aug 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 26 of 26)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"](https://mlsanity.com/paper/2309.12288/) (2023)
- [Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning](https://mlsanity.com/paper/2305.12295/) (2023)
- [Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning](https://mlsanity.com/paper/2205.09712/) (2022)
- [ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language](https://mlsanity.com/paper/2012.13048/) (2020)
- [LINC: A Neurosymbolic Approach for Logical Reasoning by Combining Language Models with First-Order Logic Provers](https://mlsanity.com/paper/2310.15164/) (2023)
- [CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text](https://mlsanity.com/paper/1908.06177/) (2019)
- [A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts](https://mlsanity.com/paper/2402.09727/) (2024)
- [ChatDB: Augmenting LLMs with Databases as Their Symbolic Memory](https://mlsanity.com/paper/2306.03901/) (2023)
- [Deductive Verification of Chain-of-Thought Reasoning](https://mlsanity.com/paper/2306.03872/) (2023)
- [Faithful Logical Reasoning via Symbolic Chain-of-Thought](https://mlsanity.com/paper/2405.18357/) (2024)
- [Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement](https://mlsanity.com/paper/2310.08559/) (2023)
- [Large Language Models can Learn Rules](https://mlsanity.com/paper/2310.07064/) (2023)
- [AR-LSAT: Investigating Analytical Reasoning of Text](https://mlsanity.com/paper/2104.06598/) (2021)
- [Logic-Driven Context Extension and Data Augmentation for Logical Reasoning of Text](https://mlsanity.com/paper/2105.03659/) (2021)
- [From LSAT: The Progress and Challenges of Complex Reasoning](https://mlsanity.com/paper/2108.00648/) (2021)
- [Statler: State-Maintaining Language Models for Embodied Reasoning](https://mlsanity.com/paper/2306.17840/) (2023)
- [Learning to Reason and Memorize with Self-Notes](https://mlsanity.com/paper/2305.00833/) (2023)
- [FaiRR: Faithful and Robust Deductive Reasoning over Natural Language](https://mlsanity.com/paper/2203.10261/) (2022)
- [Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs](https://mlsanity.com/paper/2402.11442/) (2024)
- [Empowering Working Memory for Large Language Model Agents](https://mlsanity.com/paper/2312.17259/) (2023)
- [LongHeads: Multi-Head Attention is Secretly a Long Context Processor](https://mlsanity.com/paper/2402.10685/) (2024)

## Built on by (most cited first, 17 of 17)

- [Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topics](https://mlsanity.com/paper/2505.00675/) (2025)
- [RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World Scenarios](https://mlsanity.com/paper/2412.08972/) (2024)
- [From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms](https://mlsanity.com/paper/2605.06716/) (2026)
- [MatterChat: A Multi-Modal LLM for Material Science](https://mlsanity.com/paper/2502.13107/) (2025)
- [Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression](https://mlsanity.com/paper/2505.19433/) (2025)
- [RuleRAG: Rule-Guided Retrieval-Augmented Generation with Language Models for Question Answering](https://mlsanity.com/paper/2410.22353/) (2024)
- [On the Failure of Latent State Persistence in Large Language Models](https://mlsanity.com/paper/2505.10571/) (2025)
- [MAGNET: Towards Adaptive GUI Agents with Memory-Driven Knowledge Evolution](https://mlsanity.com/paper/2601.19199/) (2026)
- [A Review of Incorporating Psychological Theories in LLMs](https://mlsanity.com/paper/2505.00003/) (2025)
- [RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Sampling](https://mlsanity.com/paper/2506.08672/) (2025)
- [Improve Rule Retrieval and Reasoning with Self-Induction and Relevance ReEstimate](https://mlsanity.com/paper/2505.10870/) (2025)
- [RIMRULE: Improving Tool-Using Language Agents via MDL-Guided Rule Learning](https://mlsanity.com/paper/2601.00086/) (2025)
- [M3Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks](https://mlsanity.com/paper/2606.05008/) (2026)
- [Track, Rank, Crack: Epistemic Working Memory Scales Multi-Hop Reasoning in Language Agents](https://mlsanity.com/paper/2607.12267/) (2026)
- [CMQCIC-Bench: A Chinese Benchmark for Evaluating Large Language Models in Medical Quality Control Indicator Calculation](https://mlsanity.com/paper/2502.11703/) (2025)
- [TVS Sidekick: Challenges and Practical Insights from Deploying Large Language Models in the Enterprise](https://mlsanity.com/paper/2509.26482/) (2025)
- [LLMs Are Not Good Strategists, Yet Memory-Enhanced Agency Boosts Reasoning](https://mlsanity.com/paper/2608.12626/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have shown remarkable reasoning performance but struggle with multi-step deductive reasoning involving a series of rule application steps, especially when rules are presented non-sequentially. Our preliminary analysis shows that while LLMs excel in single-step rule application, their performance drops significantly in multi-step scenarios due to the challenge in rule grounding. It requires anchoring the applicable rule and supporting facts at each step, amidst multiple input rules, facts, and inferred facts. To address this, we propose augmenting LLMs with external working memory and introduce a neurosymbolic framework for rule application. The memory stores facts and rules in both natural language and symbolic forms, enabling precise tracking. Utilizing this memory, our framework iteratively performs symbolic rule grounding and LLM-based rule implementation. The former matches predicates and variables of symbolic rules and facts to ground applicable rules at each step. Experiments indicate our framework's effectiveness in rule application and its robustness across various steps and settings~footnoteCode and data are available at urlhttps://github.com/SiyuanWangw/RuleApplication..