# No Training Required: Exploring Random Encoders for Sentence Classification

arXiv: https://arxiv.org/abs/1901.10444 · HTML: https://mlsanity.com/paper/1901.10444/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jan 2019.

## Built on by (most cited first, 30 of 32)

- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [State Entropy Maximization with Random Encoders for Efficient Exploration](https://mlsanity.com/paper/2102.09430/) (2021)
- [COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentences](https://mlsanity.com/paper/2106.00969/) (2021)
- [Emergence of Maps in the Memories of Blind Navigation Agents](https://mlsanity.com/paper/2301.13261/) (2023)
- [An Isotropy Analysis in the Multilingual BERT Embedding Space](https://mlsanity.com/paper/2110.04504/) (2021)
- [Mechanistic?](https://mlsanity.com/paper/2410.09087/) (2024)
- [Knowing Before Saying: LLM Representations Encode Information About Chain-of-Thought Success Before Completion](https://mlsanity.com/paper/2505.24362/) (2025)
- [Efficient and Private Federated Learning with Partially Trainable Networks](https://mlsanity.com/paper/2110.03450/) (2021)
- [Knowledge Distillation of Russian Language Models with Reduction of Vocabulary](https://mlsanity.com/paper/2205.02340/) (2022)
- [Empirical Evaluation and Theoretical Analysis for Representation Learning: A Survey](https://mlsanity.com/paper/2204.08226/) (2022)
- [Backdoor Learning for NLP: Recent Advances, Challenges, and Future Research Directions](https://mlsanity.com/paper/2302.06801/) (2023)
- [Identifying and Mitigating the Influence of the Prior Distribution in Large Language Models](https://mlsanity.com/paper/2504.12585/) (2025)
- [On the Interpretability of Deep Learning Based Models for Knowledge Tracing](https://mlsanity.com/paper/2101.11335/) (2021)
- [What's Hidden in a One-layer Randomly Weighted Transformer?](https://mlsanity.com/paper/2109.03939/) (2021)
- [Private Language Models via Truncated Laplacian Mechanism](https://mlsanity.com/paper/2410.08027/) (2024)
- [Cost-effective Deployment of BERT Models in Serverless Environment](https://mlsanity.com/paper/2103.10673/) (2021)
- [Neural network embeddings recover value dimensions from psychometric survey items on par with human data](https://mlsanity.com/paper/2509.24906/) (2025)
- [Sentence Embeddings by Ensemble Distillation](https://mlsanity.com/paper/2104.06719/) (2021)
- [Text Classification and Clustering with Annealing Soft Nearest Neighbor Loss](https://mlsanity.com/paper/2107.14597/) (2021)
- [General Cross-Architecture Distillation of Pretrained Language Models into Matrix Embeddings](https://mlsanity.com/paper/2109.08449/) (2021)
- [EigenNoise: A Contrastive Prior to Warm-Start Representations](https://mlsanity.com/paper/2205.04376/) (2022)
- [Text Classification using Graph Convolutional Networks: A Comprehensive Survey](https://mlsanity.com/paper/2410.09399/) (2024)
- [Emergent Manifold Separability during Reasoning in Large Language Models](https://mlsanity.com/paper/2602.20338/) (2026)
- [Random Embeddings and Linear Regression can Predict Protein Function](https://mlsanity.com/paper/2104.14661/) (2021)
- [Identifying stimulus-driven neural activity patterns in multi-patient intracranial recordings](https://mlsanity.com/paper/2202.01933/) (2022)
- [To Know by the Company Words Keep and What Else Lies in the Vicinity](https://mlsanity.com/paper/2205.00148/) (2022)
- [Probing of Quantitative Values in Abstractive Summarization Models](https://mlsanity.com/paper/2210.00667/) (2022)
- [Establishing Deep InfoMax as an effective self-supervised learning methodology in materials informatics](https://mlsanity.com/paper/2407.00671/) (2024)
- [Syntactic Learnability of Echo State Neural Language Models at Scale](https://mlsanity.com/paper/2503.01724/) (2025)
- [Encoded but Not Actionable: Auditing the Decode-Generate-Steer Gap in Frozen LLMs for Geometric Constraints](https://mlsanity.com/paper/2608.17843/) (2026)

## Abstract (arXiv)

We explore various methods for computing sentence representations from pre-trained word embeddings without any training, i.e., using nothing but random parameterizations. Our aim is to put sentence embeddings on more solid footing by 1) looking at how much modern sentence embeddings gain over random methods---as it turns out, surprisingly little; and by 2) providing the field with more appropriate baselines going forward---which are, as it turns out, quite strong. We also make important observations about proper experimental protocol for sentence classification evaluation, together with recommendations for future research.