mlsanityData to 13 Oct 2026

DIALEVAL: Automated Type-Theoretic Evaluation of LLM Instruction Following

arXiv 2603.03321 ↗First posted 10 Feb 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Evaluating instruction following in Large Language Models requires decomposing instructions into verifiable requirements and assessing satisfaction--tasks currently dependent on manual annotation and uniform criteria that do not align with human judgment patterns. We present DIALEVAL, a type-theoretic framework using dual LLM agents to automate instruction decomposition into typed predicates and implement type-specific satisfaction semantics. The framework enforces formal atomicity and independence constraints during automated extraction, then applies differentiated evaluation criteria--semantic equivalence for content predicates, exact precision for numerical predicates--mirroring empirically observed human assessment patterns. Extended to multi-turn dialogues through history-aware satisfaction functions, DIALEVAL enables evaluation in conversational contexts where single-turn methods fail. Validation demonstrates 90.38% accuracy (26.45% error reduction over baselines) and substantially stronger correlation with human judgment for complex instructions.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 13 of its 16 references that are in the map. First posted on arXiv 10 Feb 2026.

Builds on

13 of 16 references.

References · 13

  1. 2022Training language models to follow instructions with human feedbackcited by 15,856
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,876
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,463
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,693
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,228
  7. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,191
  8. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 745
  9. 2020Recipes for building an open-domain chatbotcited by 593
  10. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 421
  11. 2023GPTScore: Evaluate as You Desirecited by 361
  12. 2024InFoBench: Evaluating Instruction Following Ability in Large Language Modelscited by 132
1 more
  1. 2025Bot Wars Evolved: Orchestrating Competing LLMs in a Counterstrike Against Phone Scamscited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Enhancing LLM Instruction Following: An Evaluation-Driven Multi-Agentic Workflow for Prompt Instructions Optimization
  2. 2026IndicIFEval: A Benchmark for Verifiable Instruction-Following Evaluation in 14 Indic Languages
  3. 2024Benchmarking Large Language Models on Controllable Generation under Diversified Instructions
  4. 2023FollowEval: A Multi-Dimensional Benchmark for Assessing the Instruction-Following Capability of Large Language Models
  5. 2024Diverse and Fine-Grained Instruction-Following Ability Exploration with Synthetic Data
  6. 2026Do as I Say, Not as I Do: Instruction-Induction Conflict in LLMs
  7. 2025IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluation
  8. 2025An Empirical Study of LLM-as-a-Judge: How Design Choices Impact Evaluation Reliability