mlsanityData to 14 Oct 2026

Revisiting Compositional Generalization Capability of Large Language Models Considering Instruction Following Ability

arXiv 2506.15629 ↗First posted 18 Jun 2025cs.CL cs.AIcited by 10

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

In generative commonsense reasoning tasks such as CommonGen, generative large language models (LLMs) compose sentences that include all given concepts. However, when focusing on instruction-following capabilities, if a prompt specifies a concept order, LLMs must generate sentences that adhere to the specified order. To address this, we propose Ordered CommonGen, a benchmark designed to evaluate the compositional generalization and instruction-following abilities of LLMs. This benchmark measures ordered coverage to assess whether concepts are generated in the specified order, enabling a simultaneous evaluation of both abilities. We conducted a comprehensive analysis using 36 LLMs and found that, while LLMs generally understand the intent of instructions, biases toward specific concept order patterns often lead to low-diversity outputs or identical results even when the concept order is altered. Moreover, even the most instruction-compliant LLM achieved only about 75% ordered coverage, highlighting the need for improvements in both instruction-following and compositional generalization capabilities.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 67 of its 120 references that are in the map. First posted on arXiv 18 Jun 2025.

Builds on

67 of 120 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024What Goes Into a LM Acceptability Judgment? Rethinking the Impact of Frequency and Length4 Nov
  2. 2024Improving Diversity of Commonsense Generation by Large Language Models via In-Context Learning25 Apr
  3. 2024ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic20 Feb
  4. 2023Large Language Models Only Pass Primary School Exams in Indonesia: A Comprehensive Test on IndoMMLU7 Oct
  5. 2022DimonGen: Diversified Generative Commonsense Reasoning for Explaining Concept Relationships20 Dec
  6. 2022Revisiting Generative Commonsense Reasoning: A Pre-Ordering Approach26 May
  7. 2022Diversifying Content Generation for Commonsense Reasoning with Mixture of Knowledge Graph Experts14 Mar

Other references · 60

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  3. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  6. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  7. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  8. 2024A Survey on LLM-as-a-Judgecited by 1,423
  9. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  10. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,077
  11. 2018Neural Network Acceptability Judgmentscited by 958
  12. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
38 more
  1. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 824
  2. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 739
  3. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  4. 2023CMMLU: Measuring massive multitask language understanding in Chinesecited by 449
  5. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  6. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 417
  7. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  8. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 251
  9. 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 205
  10. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  11. 2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
  12. 2024Marco-o1: Towards Open Reasoning Models for Open-Ended Solutionscited by 144
  13. 2024Hello Again! LLM-powered Personalized Agent for Long-term Dialoguecited by 123
  14. 2019Mixture Models for Diverse Machine Translation: Tricks of the Tradecited by 77
  15. 2024Tell Me More! Towards Implicit User Intention Understanding of Language Model Driven Agentscited by 74
  16. 2022Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk Decodingcited by 54
  17. 2023Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Studycited by 43
  18. 2024Graph-enhanced Large Language Models in Asynchronous Plan Reasoningcited by 37
  19. 2024LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraintscited by 35
  20. 2023A Survey of the Evolution of Language Model-Based Dialogue Systems: Data, Task and Modelscited by 23
  21. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Surveycited by 23
  22. 2024ReIFE: Re-evaluating Instruction-Following Evaluationcited by 22
  23. 2024Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practicescited by 20
  24. 2023Unnatural Error Correction: GPT-4 Can Almost Perfectly Handle Unnatural Scrambled Textcited by 19
  25. 2023Flacuna: Unleashing the Problem Solving Power of Vicuna using FLAN Fine-Tuningcited by 18
  26. 2024CharPoet: A Chinese Classical Poetry Generation System Based on Token-free LLMcited by 18
  27. 2023Striking Gold in Advertising: Standardization and Exploration of Ad Text Generationcited by 17
  28. 2024LLM-based multi-agent poetry generation in non-cooperative environmentscited by 15
  29. 2024How to Make the Most of LLMs' Grammatical Knowledge for Acceptability Judgmentscited by 14
  30. 2024mbrs: A Library for Minimum Bayes Risk Decodingcited by 13
  31. 2020Generating Major Types of Chinese Classical Poetry in a Uniformed Frameworkcited by 12
  32. 2022Finding Memo: Extractive Memorization in Constrained Sequence Generation Taskscited by 12
  33. 2024MORE: Multi-mOdal REtrieval Augmented Generative Commonsense Reasoningcited by 12
  34. 2024Generating Diverse and High-Quality Texts by Minimum Bayes Risk Decodingcited by 11
  35. 2024Developing Story: Case Studies of Generative AI's Use in Journalismcited by 10
  36. 2023From Cooking Recipes to Robot Task Trees -- Improving Planning Correctness and Task Efficiency by Leveraging LLMs with a Knowledge Networkcited by 9
  37. 2025gec-metrics: A Unified Library for Grammatical Error Correction Evaluationcited by 9
  38. 2019A Hierarchical Attention Based Seq2seq Model for Chinese Lyrics Generationcited by 7

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 8.

10 papers.

  1. 2025Replay Failures as Successes: Sample-Efficient Reinforcement Learning for Instruction Followingcited by 8
  2. 2025What Drives Compositional Generalization in Visual Generative Models? The Importance of Continuous Training Objectivescited by 3
  3. 2026GlossoGen: Emergent Language in Complex Multi-Agent LLM Interactionscited by 1
  4. 2026FM SO.P: A Progressive Task Mixture Framework with Automatic Evaluation for Cross-Domain SOP Understandingcited by 0
  5. 2026Neurosymbolic Language Reasoning as Satisfiability Modulo Theorycited by 0
  6. 2026AGEL-Comp: A Neuro-Symbolic Framework for Compositional Generalization in Interactive Agentscited by 0
  7. 2026Toward Skill-Native LLMs: Skill Entropy for Benchmarking and Training Long-Horizon Reasoningcited by 0
  8. 2026Large Language Models Can Follow Instructions, But Not Many at Once: Phase Transitions in Compositional Constraint Satisfactioncited by 0
  9. 2026The Rise of Verbal Reinforcement Learningcited by 0
  10. 2026Same Scene, Different Task: Skill Alignment for Compositional Generalization in VLAscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Benchmarking Large Language Models on Controllable Generation under Diversified Instructions
  2. 2024The Mystery of Compositional Generalization in Graph-based Generative Commonsense Reasoning
  3. 2023Skills-in-Context Prompting: Unlocking Compositionality in Large Language Models
  4. 2024Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalability
  5. 2025Compositional Generalization from Learned Skills via CoT Training: A Theoretical and Structural Analysis for Reasoning
  6. 2024Towards Generalist Prompting for Large Language Models by Mental Models
  7. 2025The Instruction Gap: LLMs get lost in Following Instruction
  8. 2023FollowEval: A Multi-Dimensional Benchmark for Assessing the Instruction-Following Capability of Large Language Models