# Cognitive Modeling with Scaffolded LLMs: A Case Study of Referential Expression Generation

arXiv: https://arxiv.org/abs/2407.03805 · HTML: https://mlsanity.com/paper/2407.03805/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 48 references that are in the map.
- First posted on arXiv 4 Jul 2024.

## Builds on (most cited first, 29 of 29)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [Cognitive Architectures for Language Agents](https://mlsanity.com/paper/2309.02427/) (2023)
- [Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning](https://mlsanity.com/paper/2205.09712/) (2022)
- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [Large Language Models for Mathematical Reasoning: Progresses and Challenges](https://mlsanity.com/paper/2402.00157/) (2024)
- [Generated Knowledge Prompting for Commonsense Reasoning](https://mlsanity.com/paper/2110.08387/) (2021)
- [ART: Automatic multi-step reasoning and tool-use for large language models](https://mlsanity.com/paper/2303.09014/) (2023)
- [Generating Visual Explanations](https://mlsanity.com/paper/1603.08507/) (2016)
- [Solving Math Word Problems by Combining Language Models With Symbolic Solvers](https://mlsanity.com/paper/2304.09102/) (2023)
- [From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thought](https://mlsanity.com/paper/2306.12672/) (2023)
- [Reasoning About Pragmatics with Neural Listeners and Speakers](https://mlsanity.com/paper/1604.00562/) (2016)
- [Context-aware Captions from Context-agnostic Supervision](https://mlsanity.com/paper/1701.02870/) (2017)
- [Pragmatically Informative Image Captioning with Character-Level Inference](https://mlsanity.com/paper/1804.05417/) (2018)
- [Learning in the Rational Speech Acts Model](https://mlsanity.com/paper/1510.06807/) (2015)
- [Pragmatic Issue-Sensitive Image Captioning](https://mlsanity.com/paper/2004.14451/) (2020)
- [Know What You Don't Know: Modeling a Pragmatic Speaker that Refers to Objects of Unknown Categories](https://mlsanity.com/paper/1906.05518/) (2019)
- [Bayesian Statistical Modeling with Predictors from LLMs](https://mlsanity.com/paper/2406.09012/) (2024)
- [Evaluating Pragmatic Abilities of Image Captioners on A3DS](https://mlsanity.com/paper/2305.12777/) (2023)

## Built on by (most cited first, 2 of 2)

- [Integrating Neural and Symbolic Components in a Model of Pragmatic Question-Answering](https://mlsanity.com/paper/2506.01474/) (2025)
- [Computational models of pragmatic reasoning with flexible generation of meaning and expression alternatives](https://mlsanity.com/paper/2607.18443/) (2026)

## Abstract (arXiv)

To what extent can LLMs be used as part of a cognitive model of language generation? In this paper, we approach this question by exploring a neuro-symbolic implementation of an algorithmic cognitive model of referential expression generation by Dale & Reiter (1995). The symbolic task analysis implements the generation as an iterative procedure that scaffolds symbolic and gpt-3.5-turbo-based modules. We compare this implementation to an ablated model and a one-shot LLM-only baseline on the A3DS dataset (Tsvilodub & Franke, 2023). We find that our hybrid approach is cognitively plausible and performs well in complex contexts, while allowing for more open-ended modeling of language generation in a larger domain.