mlsanityData to 14 Oct 2026

Structured Thoughts Automaton: First Formalized Execution Model for Auto-Regressive Language Models

arXiv 2306.10196 ↗First posted 16 Jun 2023cs.CL cs.AI cs.FLcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

In recent months, Language Models (LMs) have become a part of daily discourse, with focus on OpenAI and the potential of Artificial General Intelligence (AGI). Furthermore, the leaking of LLama's weights to the public has led to an influx of innovations demonstrating the impressive capabilities of generative LMs. While we believe that AGI is still a distant goal, we recognize the potential of LMs in solving tasks such as searching complex documents, compiling reports with basic analysis, and providing assistance in problem-solving. In this paper, we propose formalizing the execution model of language models. We investigate current execution models, to find that this formalism has received little attention, and present our contribution: the first formalized execution model for LMs. We introduce a new algorithm for sampling the predictions of LMs, which we use to build a reliable and inspectable execution model. We introduce a low-level language to write "cognitive program" for this execution model. We hope to shed light on the need for execution models for LMs and encourage further research in this area.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 14 of its 35 references that are in the map. First posted on arXiv 16 Jun 2023.

Builds on

14 of 35 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023PaLM 2 Technical Report17 May
  2. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Models6 Oct

Other references · 11

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  7. 2023A Survey of Large Language Modelscited by 2,576
  8. 2023StarCoder: may the source be with you!cited by 799
  9. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  10. 2022Prompting Is Programming: A Query Language for Large Language Modelscited by 110
  11. 2019A Survey on Neural Network Language Modelscited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Designing Algorithms Empowered by Language Models: An Analytical Framework, Case Studies, and Insights
  2. 2023L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models
  3. 2025LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research
  4. 2024AI-native Memory: A Pathway from LLMs Towards AGI
  5. 2024When a language model is optimized for reasoning, does it still show embers of autoregression? An analysis of OpenAI o1
  6. 2025A Survey on Large Language Models with some Insights on their Capabilities and Limitations
  7. 2026AutoTrainess: Teaching Language Models to Improve Language Models Autonomously
  8. 2024LLMs for Mathematical Modeling: Towards Bridging the Gap between Natural and Mathematical Languages