Punching Above Their Weight: Classification-Head Fine-Tuning of Tiny Language Models (TLMs) for Verifiable Multiple-Choice Tasks
Not on the map. Why?
Abstract (arXiv)
We define Tiny Language Models (TLMs) as models below roughly 3B parameters that fit on mainstream consumer devices. We study how to adapt them for and use them on verifiable multiple-choice tasks. We compare three LoRA-based fine-tuning paradigms (label generation, gold only, and our discriminative classification head) on a unified setup across several Qwen3 models from 0.6B to 8B and five benchmarks: HellaSwag, WinoGrande, PIQA, SciQ and ARC-C. Classification-head fine-tuning reliably outperforms label generation (+2-3%) at the 0.6B and 1.7B scales. Further, TLMs fine-tuned using the discriminative method are competitive to zero-/few-shot GPT-3 (175B), PaLM (540B) and GPT-4. The performance we report for Qwen3-0.6B and Qwen3-1.7B are SOTA on HellaSwag, WinoGrande, and PIQA.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 15 of its 33 references that are in the map. First posted on arXiv 4 Jul 2026.
Builds on
15 of 33 references.
References · 15
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
- 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
- 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
- 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
- 2017Crowdsourcing Multiple Choice Science Questionscited by 765
- 2024A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthinesscited by 127
- 2025Small Models Struggle to Learn from Strong Reasonerscited by 113
- 2024Small Language Models: Survey, Measurements, and Insightscited by 112
- 2021UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmarkcited by 98
- 2024Efficient multi-prompt evaluation of LLMscited by 73
- 2025Right Answer, Wrong Score: Uncovering the Inconsistencies of LLM Evaluation in Multiple-Choice Question Answeringcited by 28
3 more
- 2024TinyLLM: A Framework for Training and Deploying Language Models at the Edge Computerscited by 9
- 2025TinyLLM: Evaluation and Optimization of Small Language Models for Agentic Tasks on Edge Devicescited by 6
- 2025Fine-Tuning Causal LLMs for Text Classification: Embedding-Based vs. Instruction-Based Approachescited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.