mlsanityData to 14 Oct 2026

Character-Aware Neural Language Models

arXiv 1508.06615 ↗First posted 26 Aug 2015cs.CL cs.NE stat.MLcited by 132

Not on the map. Why?

Abstract (arXiv)

We describe a simple neural language model that relies only on character-level inputs. Predictions are still made at the word-level. Our model employs a convolutional neural network (CNN) and a highway network over characters, whose output is given to a long short-term memory (LSTM) recurrent neural network language model (RNN-LM). On the English Penn Treebank the model is on par with the existing state-of-the-art despite having 60% fewer parameters. On languages with rich morphology (Arabic, Czech, French, German, Spanish, Russian), the model outperforms word-level/morpheme-level LSTM baselines, again with fewer parameters. The results suggest that on many languages, character inputs are sufficient for language modeling. Analysis of word representations obtained from the character composition part of the model reveals that the model is able to encode, from characters only, both semantic and orthographic information.

Key facts. Cited by 132 papers in the map (data through 14 Oct 2026). First posted on arXiv 26 Aug 2015.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 53; 2022: 27; 2023: 18; 2024: 14; 2025: 12; 2026: 8.

132 papers.

Sort:

By area:

  1. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  2. 2021Curriculum Learning: A Surveycited by 259
  3. 2021CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representationcited by 232
  4. 2021Charformer: Fast Character Transformers via Gradient-based Subword Tokenizationcited by 160
  5. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 159
  6. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 132
  7. 2021Papaya: Practical, Private, and Scalable Federated Learningcited by 90
  8. 2022Where to Begin? On the Impact of Pre-Training and Initialization in Federated Learningcited by 71
  9. 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63
  10. 2021The Neural Data Router: Adaptive Control Flow in Transformers Improves Systematic Generalizationcited by 62
  11. 2023One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Eracited by 55
  12. 2021A Survey on Green Deep Learningcited by 52
38 more
  1. 2022CodeFill: Multi-token Code Completion by Jointly Learning from Structure and Naming Sequencescited by 50
  2. 2021Robust Open-Vocabulary Translation from Visual Text Representationscited by 47
  3. 2021Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Surveycited by 46
  4. 2021With a Little Help from my Temporal Context: Multimodal Egocentric Action Recognitioncited by 41
  5. 2021Multi-view Subword Regularizationcited by 40
  6. 2022What do tokens know about their characters and how do they know it?cited by 36
  7. 2021StylePTB: A Compositional Benchmark for Fine-grained Controllable Text Style Transfercited by 34
  8. 2021AMMU : A Survey of Transformer-based Biomedical Pretrained Language Modelscited by 32
  9. 2024MrT5: Dynamic Token Merging for Efficient Byte-level Language Modelscited by 28
  10. 2021Dynamic Language Models for Continuously Evolving Contentcited by 27
  11. 2022Visualizing and Understanding Patch Interactions in Vision Transformercited by 26
  12. 2021Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-trainingcited by 22
  13. 2021Entity Linking Meets Deep Learning: Techniques and Solutionscited by 22
  14. 2023An Overview on Language Models: Recent Developments and Outlookcited by 22
  15. 2024Large Language Models Lack Understanding of Character Composition of Wordscited by 22
  16. 2023Towards Cross-Table Masked Pretraining for Web Data Miningcited by 21
  17. 2024Data Poisoning for In-context Learningcited by 21
  18. 2025Scaling Embedding Layers in Language Modelscited by 20
  19. 2022Saga: A Platform for Continuous Construction and Serving of Knowledge At Scalecited by 19
  20. 2021Neural ranking models for document retrievalcited by 16
  21. 20211xN Pattern for Pruning Convolutional Neural Networkscited by 16
  22. 2023Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translationcited by 15
  23. 2021A Survey on Understanding, Visualizations, and Explanation of Deep Neural Networkscited by 14
  24. 2024Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5cited by 14
  25. 2021Neural OCR Post-Hoc Correction of Historical Corporacited by 12
  26. 2024Model Decides How to Tokenize: Adaptive DNA Sequence Tokenization with MxDNAcited by 12
  27. 2021Embedding Symbolic Temporal Knowledge into Deep Sequential Modelscited by 11
  28. 2024From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemescited by 11
  29. 2021Hybrid quantum-classical machine learning for generative chemistry and drug designcited by 10
  30. 2025Continuous Autoregressive Language Modelscited by 10
  31. 2021Misspelling Correction with Pre-trained Contextual Language Modelcited by 9
  32. 2021Recent Trends in Named Entity Recognition (NER)cited by 9
  33. 2022Computer Science Named Entity Recognition in the Open Research Knowledge Graphcited by 9
  34. 2021Automatic Code Generation using Pre-Trained Language Modelscited by 8
  35. 2021Evaluating Neural Word Embeddings for Sanskritcited by 8
  36. 2022A Survey of Implicit Discourse Relation Recognitioncited by 8
  37. 2023Green Federated Learningcited by 8
  38. 2021Language Modelling via Learning to Rankcited by 7

50 most cited of 132 All 132 (JSON)