mlsanityData to 7 Oct 2026

Rethinking Data: Towards Better Performing Domain-Specific Small Language Models

arXiv 2503.01464 ↗First posted 3 Mar 2025cs.CL cs.AIcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Fine-tuning of Large Language Models (LLMs) for downstream tasks, performed on domain-specific data has shown significant promise. However, commercial use of such LLMs is limited by the high computational cost required for their deployment at scale. On the other hand, small Language Models (LMs) are much more cost effective but have subpar performance in a similar setup. This paper presents our approach to finetuning a small LM, that reaches high accuracy in multiple choice question answering task. We achieve this by improving data quality at each stage of the LM training pipeline. In particular, we start with data structuring resulting in extraction of compact, semantically meaningful text chunks used by a retriever. This allows more efficient knowledge digestion by the LM. Further, we improve the retrieved context by training a lightweight Chunk Re-Ranker (CRR) that generates more accurate relative relevance chunk scores. Finally, we improve the model generalization ability by merging the models fine-tuned with different parameters on different data subsets. We present detailed procedure descriptions, and corresponding experimental findings that show the improvements of each one of the proposed techniques.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 15 of its 35 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

15 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLaMA: Open and Efficient Foundation Language Models27 Feb

Other references · 14

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  3. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  4. 2023The Falcon Series of Open Language Modelscited by 526
  5. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  6. 2023RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentationcited by 253
  7. 2022Leveraging Large Language Models for Multiple Choice Question Answeringcited by 199
  8. 2023PaperQA: Retrieval-Augmented Generative Agent for Scientific Researchcited by 143
  9. 2019Unsupervised Question Answering by Cloze Translationcited by 59
  10. 2024Don't Forget to Connect! Improving RAG with Graph-based Rerankingcited by 49
  11. 2023Empower Large Language Model to Perform Better on Industrial Domain-Specific Question Answeringcited by 41
  12. 2024LLM4Rerank: LLM-based Auto-Reranking Framework for Recommendationscited by 40
2 more
  1. 2024LLMs May Perform MCQA by Selecting the Least Incorrect Optioncited by 32
  2. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2022Internet-augmented language models through few-shot prompting for open-domain question answering
  2. 2024Rethinking Generative Large Language Model Evaluation for Semantic Comprehension
  3. 2024Prompting-based Synthetic Data Generation for Few-Shot Question Answering
  4. 2024BLADE: Enhancing Black-box Large Language Models with Small Domain-Specific Models
  5. 2023Large Language Model Programs
  6. 2025Optimising Language Models for Downstream Tasks: A Post-Training Perspective
  7. 2025Towards Efficient Multi-LLM Inference: Characterization and Analysis of LLM Routing and Hierarchical Techniques
  8. 2023ToolQA: A Dataset for LLM Question Answering with External Tools