mlsanityData to 7 Oct 2026

Jump to better conclusions: SCAN both left and right

arXiv 1809.04640 ↗First posted 12 Sep 2018cs.CLcited by 29

Not on the map. Why?

Abstract (arXiv)

Lake and Baroni (2018) recently introduced the SCAN data set, which consists of simple commands paired with action sequences and is intended to test the strong generalization abilities of recurrent sequence-to-sequence models. Their initial experiments suggested that such models may fail because they lack the ability to extract systematic rules. Here, we take a closer look at SCAN and show that it does not always capture the kind of generalization that it was designed for. To mitigate this we propose a complementary dataset, which requires mapping actions back to the original commands, called NACS. We show that models that do well on SCAN do not necessarily do well on NACS, and that NACS exhibits properties more closely aligned with realistic use-cases for sequence-to-sequence models.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Sep 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 8; 2023: 5; 2024: 5; 2025: 2; 2026: 3.

29 papers.

Sort:

By area:

  1. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  2. 2021The paradox of the compositionality of natural language: a neural machine translation case studycited by 58
  3. 2021On Compositional Generalization of Neural Machine Translationcited by 34
  4. 2022Unobserved Local Structures Make Compositional Generalization Hardcited by 29
  5. 2023Neural-Logic Human-Object Interaction Detectioncited by 25
  6. 2021ReaSCAN: Compositional Reasoning in Language Groundingcited by 19
  7. 2021Inducing Transformer's Compositional Generalization Ability via Auxiliary Sequence Prediction Taskscited by 19
  8. 2021Can Transformers Jump Around Right in Natural Language? Assessing Performance Transfer from SCANcited by 14
  9. 2022Improving Systematic Generalization Through Modularity and Augmentationcited by 13
  10. 2022Benchmarking Compositionality with Formal Languagescited by 12
  11. 2022Pragmatics in Language Grounding: Phenomena, Tasks, and Modeling Approachescited by 12
  12. 2023The Validity of Evaluation Results: Assessing Concurrence Across Compositionality Benchmarkscited by 12
17 more
  1. 2022Mutual Exclusivity Training and Primitive Augmentation to Induce Compositionalitycited by 8
  2. 2023On Evaluating Multilingual Compositional Generalization with Translated Datasetscited by 6
  3. 2023Data Factors for Better Compositional Generalizationcited by 5
  4. 2025Behavioural vs. Representational Systematicity in End-to-End Models: An Opinionated Surveycited by 5
  5. 2026Seeing to Generalize: How Visual Data Corrects Binding Shortcutscited by 4
  6. 2023Improved Compositional Generalization by Generating Demonstrations for Meta-Learningcited by 3
  7. 2025The Gold Medals in an Empty Room: Diagnosing Metalinguistic Reasoning in LLMs with Camlangcited by 3
  8. 2024PizzaCommonSense: Learning to Model Commonsense Reasoning about Intermediate Steps in Cooking Recipescited by 2
  9. 2024Symbolic Autoencoding for Self-Supervised Sequence Learningcited by 2
  10. 2024Compositional learning of functions in humans and machinescited by 2
  11. 2022Measuring Alignment Bias in Neural Seq2Seq Semantic Parserscited by 1
  12. 2024Inducing Systematicity in Transformers by Attending to Structurally Quantized Embeddingscited by 1
  13. 2024DiNeR: a Large Realistic Dataset for Evaluating Compositional Generalizationcited by 1
  14. 2021Relational Weight Priors in Neural Networks for Abstract Pattern Learning and Language Modellingcited by 0
  15. 2022A Short Survey of Systematic Generalizationcited by 0
  16. 2026Towards Compositional Generalization in LLMs for Smart Contract Security: A Case Study on Reentrancy Vulnerabilitiescited by 0
  17. 2026Causally Evaluating the Learnability of Formal Language Taskscited by 0