More Human, More Efficient: Aligning Annotations with Quantized SLMs
Not on the map. Why?
Abstract (arXiv)
As Large Language Model (LLM) capabilities advance, the demand for high-quality annotation of exponentially increasing text corpora has outpaced human capacity, leading to the widespread adoption of LLMs in automatic evaluation and annotation. However, proprietary LLMs often exhibit systematic biases that diverge from human expert consensus, lacks reproducibility, and raises data privacy concerns. Our work examines the viability of finetuning a quantized Small Language Model of 1.7B parameter size on limited human-annotated data to serve as a highly aligned, deterministic evaluator and annotator. By implementing a custom, multi-dimensional rubric framework and simple augmentation and regularization techniques, the proposed approach achieves higher inter-annotator agreement (0.23 points increase in Krippendorff's α) than the best performing state-of-the-art proprietary LLM. We also demonstrate the generalizability of the proposed training pipeline on a separate emotion classification task. The results show that task-specific alignment and efficient 4-bit quantized fine-tuning provide superior open-source alternative to using proprietary models for evaluation and annotation. Our finetuning approach is publicly available at https://github.com/jylee-k/slm-judge.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 18 of its 22 references that are in the map. First posted on arXiv 1 Apr 2026.
Builds on
18 of 22 references.
References · 18
- 2023Large Language Models are not Fair Evaluatorscited by 1,050
- 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
- 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
- 2020GoEmotions: A Dataset of Fine-Grained Emotionscited by 479
- 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 354
- 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
- 2024Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programscited by 270
- 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 249
- 2024Self-Preference Bias in LLM-as-a-Judgecited by 248
- 2024Large Language Models for Data Annotation and Synthesis: A Surveycited by 201
- 2024A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthinesscited by 125
- 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
6 more
- 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 90
- 2025The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMscited by 72
- 2018RankME: Reliable Human Ratings for Natural Language Generationcited by 67
- 2025Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineeringcited by 64
- 2024LoRA Dropout as a Sparsity Regularizer for Overfitting Controlcited by 35
- 2024Enhancing elusive clues in knowledge learning by contrasting attention of language modelscited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024Rethinking Emotion Annotations in the Era of Large Language Models
- 2023Aligning Large Language Models with Human: A Survey
- 2026Attribute-Based Diagnosis of LLM Alignment with Hate Speech Annotations
- 2026When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger
- 2025From LLM-anation to LLM-orchestrator: Coordinating Small Models for Data Labeling
- 2025Users as Annotators: LLM Preference Learning from Comparison Mode
- 2026SHALA-LLM: Smartly Handling Ambiguous Labels in Aligning LLMs
- 2024Human-Instruction-Free LLM Self-Alignment with Limited Samples