SMART SLM: Structured Memory and Reasoning Transformer, A Small Language Model for Accurate Document Assistance
On the map: LLM reasoning → Reasoning benchmarks and evaluation
Abstract (arXiv)
The user of Engineering Manuals (EM) finds it difficult to read EM s because they are long, have a dense format which includes written documents, step by step procedures, and standard parameter lists for engineering equipment. Off the shelf transformers, especially compact ones, treat this material as a flat stream of tokens. This approach leads to confident but incorrect numeric answers and forces the models to memorize separate facts inefficiently. SMART (Structured Memory and Reasoning Transformer) offers a different and practical solution to the above problem. SMART structures its processing by using a hierarchical approach, and is based upon three main job categories (1) A syntax-aware Fact Extractor (Grammarian) Tree LSTM which extracts facts as subject relation object relations from EM sentences (2) A compact indexed memory MANN (Memory Augmented Neural Network) that indexes these Rational Subject Relation Objects as 384 dimensional vectors that are associated with the source of the information, and (3) A 6 layer Transformer that learns to fuse the previously retrieved facts into its generated response. The entire SMART model utilizes 45.51M parameters, which is 64% less than GPT-2 (124M) and 69% less than BERT (133M), and it achieves a 21.3% higher accuracy than GPT-2, indicating that SMART fits the data better with the least amount of processing requirements. SMART employs dual modes of inference an indexed fast path for known documents (sub-second answer times) and an indexed dynamic path assisted by RAGs for new uploads (FAISS Top 20 results with memory severed at 64 slots). In real world deployment, this framework leads to more well supported results with reduced hallucinations than comparable small transformer models.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 12 of its 16 references that are in the map. First posted on arXiv 24 Dec 2025.
Builds on
12 of 16 references.
References · 12
- 2017Attention Is All You Needcited by 60,361
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
- 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 482
- 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
- 2015Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networkscited by 306
- 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 230
- 2022Local Interpretable Model Agnostic Shap Explanations for machine learning modelscited by 3
- 2025OntoRAG: Enhancing Question-Answering through Automated Ontology Derivation from Unstructured Knowledge Basescited by 3
- 2025Contextual Graph Transformer: A Small Language Model for Enhanced Engineering Document Information Extractioncited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Small Language Models: Survey, Measurements, and Insights
- 2025Slm-mux: Orchestrating small language models for reasoning
- 2022TALM: Tool Augmented Language Models
- 2025Hierarchical Reasoning Models: Perspectives and Misconceptions
- 2025Guiding Reasoning in Small Language Models with LLM Assistance
- 2026MLPs are Hebbians: Constructing Efficient Fact-Storing MLPs for Transformers
- 2024A Survey of Transformer-based Language Models with Focus on Efficiency
- 2025From Memorization to Reasoning in the Spectrum of Loss Curvature