mlsanityData to 14 Oct 2026

Lissard: Long and Simple Sequential Reasoning Datasets

arXiv 2402.07859 ↗First posted 12 Feb 2024cs.CL cs.AIcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Language models are now capable of solving tasks that require dealing with long sequences consisting of hundreds of thousands of tokens. However, they often fail on tasks that require repetitive use of simple rules, even on sequences that are much shorter than those seen during training. For example, state-of-the-art LLMs can find common items in two lists with up to 20 items but fail when lists have 80 items. In this paper, we introduce Lissard, a benchmark comprising seven tasks whose goal is to assess the ability of models to process and generate wide-range sequence lengths, requiring repetitive procedural execution. Our evaluation of open-source (Mistral-7B and Mixtral-8x7B) and proprietary models (GPT-3.5 and GPT-4) show a consistent decline in performance across all models as the complexity of the sequence increases. The datasets and code are available at https://github.com/unicamp-dl/Lissard

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 3 of its 32 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

3 of 32 references.

References · 3

  1. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  2. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 139
  3. 2018Memorize or generalize? Searching for a compositional RNN in a haystackcited by 41

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024MLissard: Multilingual Long and Simple Sequential Reasoning Benchmarks
  2. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models
  3. 2024AQA-Bench: An Interactive Benchmark for Evaluating LLMs' Sequential Reasoning Ability
  4. 2026How Well Do LLMs Perform on the Simplest Long-Chain Reasoning Tasks: An Empirical Study on the Equivalence Class Problem
  5. 2025seqBench: A Tunable Benchmark to Quantify Sequential Reasoning Limits of LLMs
  6. 2025Evaluating LLMs' Reasoning Over Ordered Procedural Steps
  7. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion
  8. 2025InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Models