# Natural Language Processing with Small Feed-Forward Networks

arXiv: https://arxiv.org/abs/1708.00214 · HTML: https://mlsanity.com/paper/1708.00214/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 1 Aug 2017.

## Built on by (most cited first, 10 of 10)

- [Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets](https://mlsanity.com/paper/2103.12028/) (2021)
- [CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages](https://mlsanity.com/paper/2309.09400/) (2023)
- [Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM's Translation Capability](https://mlsanity.com/paper/2305.10266/) (2023)
- [GlotLID: Language Identification for Low-Resource Languages](https://mlsanity.com/paper/2310.16248/) (2023)
- [GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages](https://mlsanity.com/paper/2410.23825/) (2024)
- [GlotScript: A Resource and Tool for Low Resource Writing System Identification](https://mlsanity.com/paper/2309.13320/) (2023)
- [Dialect-robust Evaluation of Generated Text](https://mlsanity.com/paper/2211.00922/) (2022)
- [Lookup-Table Recurrent Language Models for Long Tail Speech Recognition](https://mlsanity.com/paper/2104.04552/) (2021)
- [On-Device Emoji Classifier Trained with GPT-based Data Augmentation for a Mobile Keyboard](https://mlsanity.com/paper/2411.05031/) (2024)
- [Kr'eyoLID From Language Identification Towards Language Mining](https://mlsanity.com/paper/2503.06547/) (2025)

## Abstract (arXiv)

We show that small and shallow feed-forward neural networks can achieve near state-of-the-art results on a range of unstructured and structured language processing tasks while being considerably cheaper in memory and computational requirements than deep recurrent models. Motivated by resource-constrained environments like mobile phones, we showcase simple techniques for obtaining such small neural network models, and investigate different tradeoffs when deciding how to allocate a small memory budget.