# Baby's CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Models

arXiv: https://arxiv.org/abs/2308.01684 · HTML: https://mlsanity.com/paper/2308.01684/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 18 of its 53 references that are in the map.
- First posted on arXiv 3 Aug 2023.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions](https://mlsanity.com/paper/1905.10044/) (2019)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [Generate rather than Retrieve: Large Language Models are Strong Context Generators](https://mlsanity.com/paper/2209.10063/) (2022)
- [Generated Knowledge Prompting for Commonsense Reasoning](https://mlsanity.com/paper/2110.08387/) (2021)
- [Call for Papers -- The BabyLM Challenge: Sample-efficient pretraining on a developmentally plausible corpus](https://mlsanity.com/paper/2301.11796/) (2023)
- [Pre-Training to Learn in Context](https://mlsanity.com/paper/2305.09137/) (2023)
- [Word Acquisition in Neural Language Models](https://mlsanity.com/paper/2110.02406/) (2021)
- [Language acquisition: do children and language models follow similar learning stages?](https://mlsanity.com/paper/2306.03586/) (2023)
- [A Review of Winograd Schema Challenge Datasets and Approaches](https://mlsanity.com/paper/2004.13831/) (2020)

## Built on by (most cited first, 9 of 9)

- [Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora](https://mlsanity.com/paper/2504.08165/) (2025)
- [LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering](https://mlsanity.com/paper/2412.12359/) (2024)
- [ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLM](https://mlsanity.com/paper/2506.14766/) (2025)
- [Not All Features Deserve Attention: Graph-Guided Dependency Learning for Tabular Data Generation with Language Models](https://mlsanity.com/paper/2507.18504/) (2025)
- [BERTtime Stories: Investigating the Role of Synthetic Story Data in Language Pre-training](https://mlsanity.com/paper/2410.15365/) (2024)
- [SAGE: Sparse Adaptive Guidance for Dependency-Aware Tabular Data Generation](https://mlsanity.com/paper/2604.24368/) (2026)
- [Basic Reading Distillation](https://mlsanity.com/paper/2507.19741/) (2025)
- [CLASS-IT: Conversational and Lecture-Aligned Small-Scale Instruction Tuning for BabyLMs](https://mlsanity.com/paper/2510.25364/) (2025)
- [Level Up: Defining and Exploiting Transitional Problems for Curriculum Learning](https://mlsanity.com/paper/2603.13761/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) demonstrate remarkable performance on a variety of natural language understanding (NLU) tasks, primarily due to their in-context learning ability. This ability could be applied to building babylike models, i.e. models at small scales, improving training efficiency. In this paper, we propose a "CoThought" pipeline, which efficiently trains smaller "baby" language models (BabyLMs) by leveraging the Chain of Thought prompting of LLMs. Our pipeline restructures a dataset of less than 100M in size using GPT-3.5-turbo, transforming it into task-oriented, human-readable texts that are comparable to the school texts for language learners. The BabyLM is then pretrained on this restructured dataset in a RoBERTa fashion. In evaluations across 4 benchmarks, our BabyLM outperforms the vanilla RoBERTa in 10 linguistic, NLU, and question-answering tasks by more than 3 points, showing a superior ability to extract contextual information. These results suggest that compact LMs pretrained on small, LLM-restructured data can better understand tasks and achieve improved performance.