mlsanityData to 7 Oct 2026

WRAVAL -- WRiting Assist eVALuation

arXiv 2601.03268 ↗First posted 19 Dec 2025cs.CL cs.LGcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

The emergence of Large Language Models (LLMs) has shifted language model evaluation toward reasoning and problem-solving tasks as measures of general intelligence. Small Language Models (SLMs) -- defined here as models under 10B parameters -- typically score 3-4 times lower than LLMs on these metrics. However, we demonstrate that these evaluations fail to capture SLMs' effectiveness in common industrial applications, such as tone modification tasks (e.g., funny, serious, professional). We propose an evaluation framework specifically designed to highlight SLMs' capabilities in non-reasoning tasks where predefined evaluation datasets don't exist. Our framework combines novel approaches in data generation, prompt-tuning, and LLM-based evaluation to demonstrate the potential of task-specific finetuning. This work provides practitioners with tools to effectively benchmark both SLMs and LLMs for practical applications, particularly in edge and private computing scenarios. Our implementation is available at: https://github.com/amazon-science/wraval.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 27 of its 35 references that are in the map. First posted on arXiv 19 Dec 2025.

Builds on

27 of 35 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Towards Reasoning Ability of Small Language Models17 Feb
  2. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samples8 Nov

Other references · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  7. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  8. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  9. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,332
  10. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  11. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  12. 2023Textbooks Are All You Needcited by 561
13 more
  1. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  2. 2025Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAscited by 491
  3. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  4. 2022Optimizing Prompts for Text-to-Image Generationcited by 204
  5. 2025Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Mathcited by 44
  6. 2024Fine-Tuning and Prompt Optimization: Two Great Steps that Work Better Togethercited by 38
  7. 2022EditEval: An Instruction-Based Benchmark for Text Improvementscited by 30
  8. 2025Apple Intelligence Foundation Language Models: Tech Report 2025cited by 25
  9. 2023Low-Rank Adaptation for Multilingual Summarization: An Empirical Studycited by 13
  10. 2024Enhancing the Reasoning Capabilities of Small Language Models via Solution Guidance Fine-Tuningcited by 11
  11. 2023Towards an On-device Agent for Text Rewritingcited by 7
  12. 2024Proofread: Fixes All Errors with One Tapcited by 4
  13. 2025Composing Policy Gradients and Prompt Optimization for Language Model Programscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026SLMJury: Can Small Language Models Judge as Well as Large Ones?
  2. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback
  3. 2024Enhancing LLM Evaluations: The Garbling Trick
  4. 2025Enhancing Reasoning Capabilities of Small Language Models with Blueprints and Prompt Template Search
  5. 2023Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks
  6. 2025Cross-Task Benchmarking and Evaluation of General-Purpose and Code-Specific Large Language Models ↗
  7. 2024Small Language Models: Survey, Measurements, and Insights
  8. 2023Through the Lens of Core Competency: Survey on Evaluation of Large Language Models