mlsanityData to 7 Oct 2026

Release of Pre-Trained Models for the Japanese Language

arXiv 2404.01657 ↗First posted 2 Apr 2024cs.CL cs.AI cs.CVcited by 22

Not on the map. Why?

Abstract (arXiv)

AI democratization aims to create a world in which the average person can utilize AI techniques. To achieve this goal, numerous research institutes have attempted to make their results accessible to the public. In particular, large pre-trained models trained on large-scale data have shown unprecedented potential, and their release has had a significant impact. However, most of the released models specialize in the English language, and thus, AI democratization in non-English-speaking communities is lagging significantly. To reduce this gap in AI access, we released Generative Pre-trained Transformer (GPT), Contrastive Language and Image Pre-training (CLIP), Stable Diffusion, and Hidden-unit Bidirectional Encoder Representations from Transformers (HuBERT) pre-trained in Japanese. By providing these models, users can freely interface with AI that aligns with Japanese cultural values and ensures the identity of Japanese culture, thus enhancing the democratization of AI. Additionally, experiments showed that pre-trained models specialized for Japanese can efficiently achieve high performance in Japanese tasks.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 25 of its 43 references that are in the map. First posted on arXiv 2 Apr 2024.

Builds on

25 of 43 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2017Proximal Policy Optimization Algorithmscited by 14,441
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2020Scaling Laws for Neural Language Modelscited by 6,411
  11. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  12. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
13 more
  1. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  2. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  3. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  8. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  9. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  10. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 562
  11. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  12. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  13. 2021Few-shot Learning with Multilingual Language Modelscited by 278

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 7; 2026: 5.

22 papers.

Sort:

By area:

  1. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  2. 2024PSLM: Parallel Generation of Text and Speech with LLMs for Low-Latency Spoken Dialogue Systemscited by 13
  3. 2024A Systematic Assessment of Language Models with Linguistic Minimal Pairs in Chinesecited by 13
  4. 2024Phase transition in large language models and the criticality of natural languagescited by 12
  5. 2025Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisionscited by 11
  6. 2024Why We Build Local Large Language Models: An Observational Analysis from 35 Japanese and Multilingual LLMscited by 4
  7. 2025Establishing a Scale for Kullback-Leibler Divergence in Language Models Across Various Settingscited by 3
  8. 2026Prosodic ABX: A Language-Agnostic Method for Measuring Prosodic Contrast in Speech Representationscited by 3
  9. 2024J-CHAT: Japanese Large-scale Spoken Dialogue Corpus for Spoken Dialogue Language Modelingcited by 2
  10. 2025Grapheme-Coherent Phonemic and Prosodic Annotation of Speech by Implicit and Explicit Grapheme Conditioningcited by 2
  11. 2026Adapting Methods for Domain-Specific Japanese Small LMs: Scale, Architecture, and Quantizationcited by 2
  12. 2024Development of a Large-scale Dataset of Chest Computed Tomography Reports in Japanese and a High-performance Finding Classification Modelcited by 1
10 more
  1. 2025Harnessing PDF Data for Improving Japanese Large Multimodal Modelscited by 1
  2. 2025Investigating the Representation of Backchannels and Fillers in Fine-tuned Language Modelscited by 1
  3. 2026Ebisu: Benchmarking Large Language Models in Japanese Financecited by 1
  4. 2026Syntactically-guided Information Maintenance in Sentence Comprehensioncited by 1
  5. 2024Pretraining and Updates of Domain-Specific LLM: A Case Study in the Japanese Business Domaincited by 0
  6. 2024Image Score: Learning and Evaluating Human Preferences for Mercari Searchcited by 0
  7. 2024Interactive DualChecker for Mitigating Hallucinations in Distilling Large Language Modelscited by 0
  8. 2025BLiSS 1.0: Evaluating Bilingual Learner Competence in Second Language Small Language Modelscited by 0
  9. 2025WAON: A Large-Scale Japanese Image-Text Dataset for Cultural Adaptation in Contrastive Vision-Language Modelscited by 0
  10. 2026Improving Large-Scale Weakly Supervised ASR by Filtering and Selectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Transformer-Based Language Models Across Domain Verticals: Architectures, Applications and Critical Assessment
  2. 2023PASTA: Pretrained Action-State Transformer Agents
  3. 2022The Myth of Culturally Agnostic AI Models
  4. 2024Generative AI in Vision: A Survey on Models, Metrics and Applications
  5. 2023The Curse of Recursion: Training on Generated Data Makes Models Forget
  6. 2025Generative AI in Depth: A Survey of Recent Advances, Model Variants, and Real-World Applications
  7. 2025GANji: A Framework for Introductory AI Image Generation
  8. 2024JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluation