mlsanityData to 14 Oct 2026

Technical Report: Small Language Model for Japanese Clinical and Medicine

arXiv 2412.16423 ↗First posted 21 Dec 2024cs.CL cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

This report presents a small language model (SLM) for Japanese clinical and medicine, named NCVC-slm-1. This 1B parameters model was trained using Japanese text classified to be of high-quality. Moreover, NCVC-slm-1 was augmented with respect to clinical and medicine content that includes the variety of diseases, drugs, and examinations. Using a carefully designed pre-processing, a specialized morphological analyzer and tokenizer, this small and light-weight model performed not only to generate text but also indicated the feasibility of understanding clinical and medicine text. In comparison to other large language models, a fine-tuning NCVC-slm-1 demonstrated the highest scores on 6 tasks of total 8 on JMED-LLM. According to this result, SLM indicated the feasibility of performing several downstream tasks in the field of clinical and medicine. Hopefully, NCVC-slm-1 will be contributed to develop and accelerate the field of clinical and medicine for a bright future.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 42 of its 71 references that are in the map. First posted on arXiv 21 Dec 2024.

Builds on

42 of 71 references.

References · 42

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2016Layer Normalizationcited by 5,669
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  10. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  11. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  12. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
30 more
  1. 2023Qwen Technical Reportcited by 3,344
  2. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  3. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,898
  4. 2024Qwen2 Technical Reportcited by 2,120
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  6. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  8. 2017Decoupled Weight Decay Regularizationcited by 1,831
  9. 2019Root Mean Square Layer Normalizationcited by 1,480
  10. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  11. 2023PaLM 2 Technical Reportcited by 1,162
  12. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 950
  13. 2020On Layer Normalization in the Transformer Architecturecited by 922
  14. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  15. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  16. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  17. 2023Textbooks Are All You Needcited by 569
  18. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  19. 2023The Falcon Series of Open Language Modelscited by 535
  20. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 356
  21. 2023MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Datacited by 286
  22. 2023MEDITRON-70B: Scaling Medical Pretraining for Large Language Modelscited by 274
  23. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  24. 2024Capabilities of Gemini Models in Medicinecited by 237
  25. 2019A Study of BFLOAT16 for Deep Learning Trainingcited by 232
  26. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  27. 2023NEFTune: Noisy Embeddings Improve Instruction Finetuningcited by 100
  28. 2023Evaluating GPT-4 and ChatGPT on Japanese Medical Licensing Examinationscited by 63
  29. 2023Spike No More: Stabilizing the Pre-training of Large Language Modelscited by 52
  30. 2024Me LLaMA: Foundation Large Language Models for Medical Applicationscited by 10

Built on by

Not cited yet.