mlsanityData to 14 Oct 2026

Llamion Technical Report

arXiv 2605.25676 ↗First posted 25 May 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We release Llamion, a family of 14B-parameter open-weight language models obtained by transforming Orion-14B into the standardized Llama-family architecture. The transformation is performed by Efficient Knowledge Preservation for Transformation (KEPT), a recipe that combines (i) Normal Parameter Mapping (NPM) for unchanged modules, (ii) Optimized Parameter Mapping (OPM), a training-free LayerNorm-to-RMSNorm initialization we prove optimal under the near-zero-mean activation regime induced by weight decay, and (iii) Cross-architecture Knowledge Distillation (XKD), an equal-size frozen-teacher distillation that aligns the converted model's outputs with the source model's on any reasonable input distribution. Llamion recovers Orion's behaviour on H6, MT-Bench, and KoMMLU with only ~123M tokens on a single A100 in four days; Llamion-Base reaches 66.87% on KoMMLU, exceeding the next-best entry of the Open Ko LLM Leaderboard by >7.0 absolute points at submission time. Capabilities entirely absent from the transfer corpus (Python programming and 200K-token context handling) survive the architectural transition intact. We release three checkpoints (Base, Chat, LongChat) that load with trust_remote_code=False in the Hugging Face Transformers library.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 28 of its 40 references that are in the map. First posted on arXiv 25 May 2026.

Builds on

28 of 40 references.

References · 28

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2016Layer Normalizationcited by 5,669
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  10. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  11. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
16 more
  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  2. 2020Knowledge Distillation: A Surveycited by 1,603
  3. 2019Root Mean Square Layer Normalizationcited by 1,480
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  5. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 722
  6. 2017Deep Mutual Learningcited by 628
  7. 2013Do Deep Nets Really Need to be Deep?cited by 474
  8. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 209
  9. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  10. 2023Scaling Laws of RoPE-based Extrapolationcited by 125
  11. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 122
  12. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 101
  13. 2024Open Ko-LLM Leaderboard: Evaluating Large Language Models in Korean with Ko-H5 Benchmarkcited by 23
  14. 2023A Technical Report for Polyglot-Ko: Open-Source Large-Scale Korean Language Modelscited by 22
  15. 2023Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspectivecited by 17
  16. 2023Preventing Catastrophic Forgetting in Continual Learning of New Natural Language Taskscited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Zebra-Llama: Towards Extremely Efficient Hybrid Models
  2. 2023Accelerating LLaMA Inference by Enabling Intermediate Layer Decoding via Instruction Tuning with LITE
  3. 2025Leveraging the true depth of LLMs
  4. 2025Evolution of meta's llama models and parameter-efficient fine-tuning of large language models: a survey
  5. 2023LLaMA: Open and Efficient Foundation Language Models
  6. 2026Llamas on the Web: Memory-Efficient, Performance-Portable, and Multi-Precision LLM Inference with WebGPU
  7. 2024LLM-NEO: Parameter Efficient Knowledge Distillation for Large Language Models
  8. 2026Measuring Maximum Activations in Open Large Language Models