mlsanityData to 14 Oct 2026

Meeseeks: A Feedback-Driven, Iterative Self-Correction Benchmark evaluating LLMs' Instruction Following Capability

arXiv 2504.21625 ↗First posted 30 Apr 2025cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

The capability to precisely adhere to instructions is a cornerstone for Large Language Models (LLMs) to function as dependable agents in real-world scenarios. However, confronted with complex prompts, LLMs frequently encounter difficulties in fulfilling all specified requirements within a single response. Drawing inspiration from recent advancements in Chain-of-Thought (CoT) prompting and self-correction methodologies, we introduce Meeseeks (The name is inspired by Mr. Meeseeks from "Rick and Morty," a character renowned for efficiently accomplishing assigned tasks. See: https://en.wikipedia.org/wiki/Mr._Meeseeks), a fully automated iterative instruction-following benchmark equipped with an integrated feedback mechanism. Meeseeks identifies erroneous components in model responses and provides corresponding feedback accurately, thereby iteratively guiding the model toward self-correction. The dataset contains over 700 curated instances annotated by 32 distinct capability tags in Chinese and English. Extensive experimental results reveal that different state-of-the-art commercial and open-source LLMs exhibit vastly disparate performance, and even after 20 turns of iterative feedback-driven self-correction, nearly all models demonstrate suboptimal performance. We conducted comprehensive analysis from both macro and instance levels, uncovering numerous common issues prevalent in current state-of-the-art models, as well as several counterintuitive phenomena. We've open-sourced our work on https://github.com/ADoublLEN/Meeseeks.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 40 of its 104 references that are in the map. First posted on arXiv 30 Apr 2025.

Builds on

40 of 104 references.

References · 40

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2023A Survey of Large Language Modelscited by 2,576
  3. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  4. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  5. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  6. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  7. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 688
  8. 2023Instruction Tuning for Large Language Models: A Surveycited by 623
  9. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  10. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 341
  11. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 325
  12. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
28 more
  1. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 222
  2. 2023The Capacity for Moral Self-Correction in Large Language Modelscited by 173
  3. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 160
  4. 2024Benchmarking Complex Instruction-Following with Multiple Constraints Compositioncited by 152
  5. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 135
  6. 2023FollowBench: A Multi-level Fine-grained Constraints Following Benchmark for Large Language Modelscited by 117
  7. 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 102
  8. 2023CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Modelscited by 93
  9. 2025HalluLens: LLM Hallucination Benchmarkcited by 92
  10. 2023COLLIE: Systematic Construction of Constrained Text Generation Taskscited by 70
  11. 2023Can Large Language Models Understand Real-World Complex Instructions?cited by 69
  12. 2023AlignBench: Benchmarking Chinese Alignment of Large Language Modelscited by 68
  13. 2023Towards Better Chain-of-Thought Prompting Strategies: A Surveycited by 62
  14. 2023Automating Customer Service using LangChain: Building custom open-source GPT Chatbot for organizationscited by 61
  15. 2024From Tokens to Words: On the Inner Lexicon of LLMscited by 48
  16. 2024Large Language Models have Intrinsic Self-Correction Abilitycited by 46
  17. 2024Financial Statement Analysis with Large Language Modelscited by 31
  18. 2024From Language Models over Tokens to Language Models over Characterscited by 31
  19. 2025StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction Followingcited by 29
  20. 2023Making Large Language Models Better Data Creatorscited by 28
  21. 2024HaluEval-Wild: Evaluating Hallucinations of Language Models in the Wildcited by 27
  22. 2024MedVH: Towards Systematic Evaluation of Hallucination for Large Vision Language Models in the Medical Contextcited by 20
  23. 2025Instruction-Following Pruning for Large Language Modelscited by 17
  24. 2023Parrot: Enhancing Multi-Turn Instruction Following for Large Language Modelscited by 16
  25. 2024On the Intrinsic Self-Correction Capability of LLMs: Uncertainty and Latent Conceptcited by 16
  26. 2024DiaHalu: A Dialogue-level Hallucination Evaluation Benchmark for Large Language Modelscited by 15
  27. 2024Knowledge of Pretrained Language Models on Surface Information of Tokenscited by 5
  28. 2023Numerical Reasoning for Financial Reportscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025LongCat-Flash Technical Reportcited by 58
  2. 2026SEQUOR: A Multi-Turn Benchmark for Realistic Constraint Followingcited by 6
  3. 2026From Static Benchmarks to Dynamic Protocol: Agent-Centric Text Anomaly Detection for Evaluating LLM Reasoningcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Self-Judge: Selective Instruction Following with Alignment Self-Evaluation
  2. 2024Confidence v.s. Critique: A Decomposition of Self-Correction Capability for LLMs
  3. 2023FollowEval: A Multi-Dimensional Benchmark for Assessing the Instruction-Following Capability of Large Language Models
  4. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Following
  5. 2024Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Models
  6. 2026Reinforcing Step-level Reasoning for Effective Self-Correction in LLMs
  7. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies
  8. 2025Can LLMs Correct Themselves? A Benchmark of Self-Correction in LLMs