mlsanityData to 14 Oct 2026

MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning

arXiv 2205.00445 ↗First posted 1 May 2022cs.CL cs.AIcited by 156

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Huge language models (LMs) have ushered in a new era for AI, serving as a gateway to natural-language-based knowledge tasks. Although an essential element of modern AI, LMs are also inherently limited in a number of ways. We discuss these limitations and how they can be avoided by adopting a systems approach. Conceptualizing the challenge as one that involves knowledge and reasoning in addition to linguistic processing, we define a flexible architecture with multiple neural models, complemented by discrete knowledge and reasoning modules. We describe this neuro-symbolic architecture, dubbed the Modular Reasoning, Knowledge and Language (MRKL, pronounced "miracle") system, some of the technical challenges in implementing it, and Jurassic-X, AI21 Labs' MRKL system implementation.

Key facts. Cited by 156 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 20 of its 28 references that are in the map. First posted on arXiv 1 May 2022.

Builds on

20 of 28 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2019Language Models as Knowledge Bases?cited by 1,936
  10. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  11. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  12. 2019How Can We Know What Language Models Know?cited by 959
8 more
  1. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  2. 2021MetaICL: Learning to Learn In Contextcited by 454
  3. 2021Sustainable AI: Environmental Implications, Challenges and Opportunitiescited by 242
  4. 2021ExT5: Towards Extreme Multi-Task Scaling for Transfer Learningcited by 161
  5. 2021Understanding the Capabilities, Limitations, and Societal Impact of Large Language Modelscited by 160
  6. 2020Break It Down: A Question Understanding Benchmarkcited by 131
  7. 2022TimeLMs: Diachronic Language Models from Twittercited by 131
  8. 2020The Cost of Training NLP Models: A Concise Overviewcited by 98

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 28; 2024: 21; 2025: 24; 2026: 80.

156 papers.

Sort:

By area:

  1. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  3. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  4. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
  5. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
  6. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 280
  7. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  8. 2024Tool Learning with Large Language Models: A Surveycited by 223
  9. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 215
  10. 2025Survey on Evaluation of LLM-based Agentscited by 204
  11. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  12. 2022Can Foundation Models Wrangle Your Data?cited by 175
38 more
  1. 2023PaperQA: Retrieval-Augmented Generative Agent for Scientific Researchcited by 149
  2. 2024Language agents achieve superhuman synthesis of scientific knowledgecited by 137
  3. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 135
  4. 2024Exploring Large Language Model based Intelligent Agents: Definitions, Methods, and Prospectscited by 132
  5. 2023From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thoughtcited by 119
  6. 2024A Review of Large Language Models and Autonomous Agents in Chemistrycited by 99
  7. 2022Successive Prompting for Decomposing Complex Questionscited by 94
  8. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  9. 2023Conversational Health Agents: A Personalized LLM-Powered Agent Frameworkcited by 92
  10. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 89
  11. 202314 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathoncited by 86
  12. 2023Large Language Models and Knowledge Graphs: Opportunities and Challengescited by 71
  13. 2023Ignore This Title and HackAPrompt: Exposing Systemic Vulnerabilities of LLMs through a Global Scale Prompt Hacking Competitioncited by 68
  14. 2022TIARA: Multi-grained Retrieval for Robust Question Answering over Large Knowledge Basescited by 59
  15. 2025TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systemscited by 59
  16. 2025MIB: A Mechanistic Interpretability Benchmarkcited by 56
  17. 2024LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editingcited by 47
  18. 2026Skill Retrieval Augmentation for Agentic AIcited by 44
  19. 2023GPT-4 Can't Reasoncited by 40
  20. 2024Are large language models superhuman chemists?cited by 34
  21. 2024CACTUS: Chemistry Agent Connecting Tool-Usage to Sciencecited by 33
  22. 2025MCP-Zero: Active Tool Discovery for Autonomous LLM Agentscited by 33
  23. 2023An In-depth Survey of Large Language Model-based Artificial Intelligence Agentscited by 31
  24. 2026AgentSentry: Mitigating Indirect Prompt Injection in LLM Agents via Temporal Causal Diagnostics and Context Purificationcited by 29
  25. 2025SciSciGPT: Advancing Human-AI Collaboration in the Science of Sciencecited by 27
  26. 2024Multi-Agent Causal Discovery Using Large Language Modelscited by 26
  27. 2024A Trembling House of Cards? Mapping Adversarial Attacks against Language Agentscited by 22
  28. 2024Efficient and Scalable Estimation of Tool Representations in Vector Spacecited by 22
  29. 2025MDCrow: Automating Molecular Dynamics Workflows with Large Language Modelscited by 22
  30. 2023Can neural networks do arithmetic? A survey on the elementary numerical skills of state-of-the-art deep learning modelscited by 18
  31. 2025Automatic Red Teaming LLM-based Agents with Model Context Protocol Toolscited by 18
  32. 2023ChatMOF: An Autonomous AI System for Predicting and Generating Metal-Organic Frameworkscited by 17
  33. 2023Controlling risks of AI in chemical science with agentscited by 17
  34. 2023How Generative AI models such as ChatGPT can be (Mis)Used in SPC Practice, Education, and Research? An Exploratory Studycited by 16
  35. 2026Agent-Sentry: Bounding LLM Agents via Execution Provenancecited by 15
  36. 2025Large Physics Models: Towards a collaborative approach with Large Language Models and Foundation Modelscited by 14
  37. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 14
  38. 2023Transformers and Large Language Models for Chemistry and Drug Discoverycited by 13

50 most cited of 156 All 156 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Neuro-Symbolic Artificial Intelligence: Towards Improving the Reasoning Abilities of Large Language Models
  2. 2023Enabling High-Level Machine Reasoning with Cognitive Neuro-Symbolic Systems
  3. 2022Is Neuro-Symbolic AI Meeting its Promise in Natural Language Processing? A Structured Review
  4. 2024Reliable Reasoning Beyond Natural Language
  5. 2026UFAL-CUNI at SemEval-2026 Task 11: An Efficient Modular Neuro-symbolic Method for Syllogistic Reasoning
  6. 2023Brain in a Vat: On Missing Pieces Towards Artificial General Intelligence in Large Language Models
  7. 2024From Linguistic Giants to Sensory Maestros: A Survey on Cross-Modal Reasoning with Large Language Models
  8. 2025Logical Reasoning in Large Language Models: A Survey