mlsanityData to 7 Oct 2026

MLissard: Multilingual Long and Simple Sequential Reasoning Benchmarks

arXiv 2410.06396 ↗First posted 8 Oct 2024cs.CLcited by 1

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Language models are now capable of solving tasks that require dealing with long sequences consisting of hundreds of thousands of tokens. However, they often fail on tasks that require repetitive use of simple rules, even on sequences that are much shorter than those seen during training. For example, state-of-the-art LLMs can find common items in two lists with up to 20 items but fail when lists have 80 items. In this paper, we introduce MLissard, a multilingual benchmark designed to evaluate models' abilities to process and generate texts of varied lengths and offers a mechanism for controlling sequence complexity. Our evaluation of open-source and proprietary models show a consistent decline in performance across all models and languages as the complexity of the sequence increases. Surprisingly, the use of in-context examples in languages other than English helps increase extrapolation performance significantly. The datasets and code are available at https://github.com/unicamp-dl/Lissard

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 15 of its 33 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

15 of 33 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Models30 Nov

Other references · 14

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  4. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  5. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  6. 2023MEGA: Multilingual Evaluation of Generative AIcited by 291
  7. 2022Neural Networks and the Chomsky Hierarchycited by 214
  8. 2023What Algorithms can Transformers Learn? A Study in Length Generalizationcited by 198
  9. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 136
  10. 2018Memorize or generalize? Searching for a compositional RNN in a haystackcited by 41
  11. 2023Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encodingcited by 32
  12. 2021Symbolic Brittleness in Sequence Models: on Systematic Generalization in Symbolic Mathematicscited by 27
2 more
  1. 2023Functional Interpolation for Relative Positions Improves Long Context Transformerscited by 18
  2. 2023Exploring Transformer Extrapolationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Uncovering Uncontrolled Repetition through Residual Stream Dynamicscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Lissard: Long and Simple Sequential Reasoning Datasets
  2. 2024AQA-Bench: An Interactive Benchmark for Evaluating LLMs' Sequential Reasoning Ability
  3. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models
  4. 2025InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Models
  5. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion
  6. 2025LongReasonArena: A Long Reasoning Benchmark for Large Language Models
  7. 2024metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models
  8. 2023Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance