mlsanityData to 7 Oct 2026

Did You Forget What I Asked? Prospective Memory Failures in Large Language Models

arXiv 2603.23530 ↗First posted 7 Mar 2026cs.CL cs.AI cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Large language models often fail to satisfy formatting instructions when they must simultaneously perform demanding tasks. We study this behaviour through a prospective memory inspired lens from cognitive psychology, using a controlled paradigm that combines verifiable formatting constraints with benchmark tasks of increasing complexity. Across three model families and over 8,000 prompts, compliance drops by 2-21% under concurrent task load. Vulnerability is highly type-dependent: terminal constraints (requiring action at the response boundary) degrade most, with drops up to 50%, while avoidance constraints remain comparatively robust. A salience-enhanced format (explicit instruction framing plus a trailing reminder) recovers much of the lost compliance, restoring performance to 90-100% in many settings. Interference is bidirectional: formatting constraints can also reduce task accuracy, with one model's GSM8K accuracy dropping from 93% to 27%. In additional stacking experiments, joint compliance declines sharply as constraints accumulate. All results use deterministic programmatic checkers without an LLM-as-judge component on publicly available datasets.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 16 of its 30 references that are in the map. First posted on arXiv 7 Mar 2026.

Builds on

16 of 30 references.

References · 16

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  5. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  6. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  7. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  8. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  9. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 441
  10. 2022Using cognitive psychology to understand GPT-3cited by 389
  11. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  12. 2024Certifiably Robust RAG against Retrieval Corruptioncited by 144
4 more
  1. 2022Capturing Failures of Large Language Models via Human Cognitive Biasescited by 97
  2. 2024Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templatescited by 94
  3. 2025IHEval: Evaluating Language Models on Following the Instruction Hierarchycited by 48
  4. 2025AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarioscited by 44

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.