# A Hierarchical Neural Autoencoder for Paragraphs and Documents

arXiv: https://arxiv.org/abs/1506.01057 · HTML: https://mlsanity.com/paper/1506.01057/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jun 2015.

## Built on by (most cited first, 26 of 26)

- [BookSum: A Collection of Datasets for Long-form Narrative Summarization](https://mlsanity.com/paper/2105.08209/) (2021)
- [A Comprehensive Survey of Scene Graphs: Generation and Application](https://mlsanity.com/paper/2104.01111/) (2021)
- [Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence](https://mlsanity.com/paper/2105.08963/) (2021)
- [Distractor Generation in Multiple-Choice Tasks: A Survey of Methods, Datasets, and Evaluation](https://mlsanity.com/paper/2402.01512/) (2024)
- [Classical Planning in Deep Latent Space](https://mlsanity.com/paper/2107.00110/) (2021)
- [Auto-Encoding Score Distribution Regression for Action Quality Assessment](https://mlsanity.com/paper/2111.11029/) (2021)
- [DiscoDVT: Generating Long Text with Discourse-Aware Discrete Variational Transformer](https://mlsanity.com/paper/2110.05999/) (2021)
- [ProTIP: Progressive Tool Retrieval Improves Planning](https://mlsanity.com/paper/2312.10332/) (2023)
- [Exploring the Latent Space of Autoencoders with Interventional Assays](https://mlsanity.com/paper/2106.16091/) (2021)
- [Video and Text Matching with Conditioned Embeddings](https://mlsanity.com/paper/2110.11298/) (2021)
- [StoryTrans: Non-Parallel Story Author-Style Transfer with Discourse Representations and Content Enhancing](https://mlsanity.com/paper/2208.13423/) (2022)
- [Keywords and Instances: A Hierarchical Contrastive Learning Framework Unifying Hybrid Granularities for Text Generation](https://mlsanity.com/paper/2205.13346/) (2022)
- [Generating Coherent Narratives by Learning Dynamic and Discrete Entity States with a Contrastive Framework](https://mlsanity.com/paper/2208.03985/) (2022)
- [Good Things Come in Pairs: Paired Autoencoders for Inverse Problems](https://mlsanity.com/paper/2505.06549/) (2025)
- [Few-Shot Learning of an Interleaved Text Summarization Model by Pretraining with Synthetic Data](https://mlsanity.com/paper/2103.05131/) (2021)
- [Autoencoders as Tools for Program Synthesis](https://mlsanity.com/paper/2108.07129/) (2021)
- [Implicit N-grams Induced by Recurrence](https://mlsanity.com/paper/2205.02724/) (2022)
- [Language Model Prompt Selection via Simulation Optimization](https://mlsanity.com/paper/2404.08164/) (2024)
- [Dominant motion identification of multi-particle system using deep learning from video](https://mlsanity.com/paper/2104.12722/) (2021)
- [Document-aware Positional Encoding and Linguistic-guided Encoding for Abstractive Multi-document Summarization](https://mlsanity.com/paper/2209.05929/) (2022)
- [A Cascaded Architecture for Extractive Summarization of Multimedia Content via Audio-to-Text Alignment](https://mlsanity.com/paper/2504.06275/) (2025)
- [Toward the Understanding of Deep Text Matching Models for Information Retrieval](https://mlsanity.com/paper/2108.07081/) (2021)
- [Unsupervised Inference of Data-Driven Discourse Structures using a Tree Auto-Encoder](https://mlsanity.com/paper/2210.09559/) (2022)
- [Adversarial Conversational Shaping for Intelligent Agents](https://mlsanity.com/paper/2307.11785/) (2023)
- [Unsupervised Candidate Answer Extraction through Differentiable Masker-Reconstructor Model](https://mlsanity.com/paper/2310.13106/) (2023)
- [Harnessing Structured Knowledge: A Concept Map-Based Approach for High-Quality Multiple Choice Question Generation with Effective Distractors](https://mlsanity.com/paper/2505.02850/) (2025)

## Abstract (arXiv)

Natural language generation of coherent long texts like paragraphs or longer documents is a challenging problem for recurrent networks models. In this paper, we explore an important step toward this generation task: training an LSTM (Long-short term memory) auto-encoder to preserve and reconstruct multi-sentence paragraphs. We introduce an LSTM model that hierarchically builds an embedding for a paragraph from embeddings for sentences and words, then decodes this embedding to reconstruct the original paragraph. We evaluate the reconstructed paragraph using standard metrics like ROUGE and Entity Grid, showing that neural models are able to encode texts in a way that preserve syntactic, semantic, and discourse coherence. While only a first step toward generating coherent text units from neural models, our work has the potential to significantly impact natural language generation and summarizationfootnoteCode for the three models described in this paper can be found at www.stanford.edu/~jiweil/ .