mlsanityData to 14 Oct 2026

H2O-Danube3 Technical Report

arXiv 2407.09276 ↗First posted 12 Jul 2024cs.CL cs.LGcited by 7

Not on the map. Why?

Abstract (arXiv)

We present H2O-Danube3, a series of small language models consisting of H2O-Danube3-4B, trained on 6T tokens and H2O-Danube3-500M, trained on 4T tokens. Our models are pre-trained on high quality Web data consisting of primarily English tokens in three stages with different data mixes before final supervised tuning for chat version. The models exhibit highly competitive metrics across a multitude of academic, chat, and fine-tuning benchmarks. Thanks to its compact architecture, H2O-Danube3 can be efficiently run on a modern smartphone, enabling local inference and rapid processing capabilities even on mobile devices. We make all models openly available under Apache 2.0 license further democratizing LLMs to a wider audience economically.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 12 of its 15 references that are in the map. First posted on arXiv 12 Jul 2024.

Builds on

12 of 15 references.

References · 12

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  5. 2023Qwen Technical Reportcited by 3,344
  6. 2023Mistral 7Bcited by 3,164
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  8. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  11. 2024TinyLlama: An Open-Source Small Language Modelcited by 618
  12. 2024MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Casescited by 223

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 2.

7 papers.

  1. 2024Programming Refusal with Conditional Activation Steeringcited by 206
  2. 2025Advantage-Guided Distillation for Preference Alignment in Small Language Modelscited by 5
  3. 2026Benchmarking Trustworthiness of SLMs: Pre-trained vs. Compressedcited by 2
  4. 2025Bridging the Digital Divide: Small Language Models as a Pathway for Physics and Photonics Education in Underdeveloped Regionscited by 0
  5. 2025Preference Distillation via Value based Reinforcement Learningcited by 0
  6. 2025GEP: A GCG-Based method for extracting personally identifiable information from chatbots built on small language modelscited by 0
  7. 2026SCAPE: Accurate and Efficient LLM Training with Extreme Sparse Communicationcited by 0