mlsanityData to 14 Oct 2026

Apertus LLM Family Expansion via Distillation and Quantization

arXiv 2605.29128 ↗First posted 27 May 2026cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

The wide adoption of LLMs has led to their use in great variety of applications and scenarios, such as chatbot assistants and data annotation, creating the need for the models to satisfy certain budget and hardware constraints. This has led to the trend of LLMs being released in batches consisting of similar models of various sizes for the family of models to adhere to as wide of a range of constraints as possible. In this paper, we validate distillation and quantization as a cost-effective way to expand model families to new sizes and hardware formats. Based on the open-recipe Apertus 8B LLM, we produce Apertus-v1.1 - a distilled family of models with up to 4B parameters trained on 1.7T permissive license tokens. We demonstrate cost-efficiency and strong accuracy performance of our approach for covering large ranges of hardware and systems requirements.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 16 of its 23 references that are in the map. First posted on arXiv 27 May 2026.

Builds on

16 of 23 references.

References · 16

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  7. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  8. 2020XCOPA: A Multilingual Dataset for Causal Commonsense Reasoningcited by 333
  9. 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 264
  10. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223
  11. 2013Estimating or Propagating Gradients Through Stochastic Neuronscited by 55
  12. 2024The AdEMAMix Optimizer: Better, Faster, Oldercited by 54
4 more
  1. 2024Pre-training Distillation for Large Language Models: A Design Space Explorationcited by 27
  2. 2026Quantization-Aware Distillation for NVFP4 Inference Accuracy Recoverycited by 22
  3. 2025Unifying Block-wise PTQ and Distillation-based QAT for Progressive Quantization toward 2-bit Instruction-Tuned LLMscited by 8
  4. 2024Deriving Activation Functions Using Integrationcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Project Greenhouse: Progress Toward Fully Open and Sovereign Agentic Searchcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From Bits to Chips: An LLM-based Hardware-Aware Quantization Agent for Streamlined Deployment of LLMs
  2. 2026QATFactory: A Versatile, Deployment-Aligned Framework for Quantization-aware Training and Distillation of LLMs
  3. 2024New Solutions on LLM Acceleration, Optimization, and Application
  4. 2024FBI-LLM: Scaling Up Fully Binarized LLMs from Scratch via Autoregressive Distillation
  5. 2023Efficient LLM Inference on CPUs
  6. 2026LLM4RTL: Tool-Assisted LLM for RTL Generation
  7. 2026BaseRT: Best-in-Class LLM Inference on Apple Silicon via Native Metal
  8. 2025Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective