mlsanityData to 7 Oct 2026

Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space

arXiv 2004.04092 ↗First posted 5 Apr 2020cs.CL cs.LG stat.MLcited by 114

Not on the map. Why?

Abstract (arXiv)

When trained effectively, the Variational Autoencoder (VAE) can be both a powerful generative model and an effective representation learning framework for natural language. In this paper, we propose the first large-scale language VAE model, Optimus. A universal latent embedding space for sentences is first pre-trained on large text corpus, and then fine-tuned for various language generation and understanding tasks. Compared with GPT-2, Optimus enables guided language generation from an abstract level using the latent vectors. Compared with BERT, Optimus can generalize better on low-resource language understanding tasks due to the smooth latent space structure. Extensive experimental results on a wide range of language tasks demonstrate the effectiveness of Optimus. It achieves new state-of-the-art on VAE language modeling benchmarks. We hope that our first pre-trained big VAE language model itself and results can help the NLP community renew the interests of deep generative models in the era of large-scale pre-training, and make these principled methods more practical.

Key facts. Cited by 114 papers in the map (data through 7 Oct 2026). First posted on arXiv 5 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 22; 2022: 35; 2023: 22; 2024: 19; 2025: 11; 2026: 4.

114 papers.

Sort:

By area:

  1. 2021Score-based Generative Modeling in Latent Spacecited by 634
  2. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  3. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Modelcited by 199
  4. 2022RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encodercited by 118
  5. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  6. 2021D2C: Diffusion-Denoising Models for Few-shot Conditional Generationcited by 96
  7. 2020Dialogue Response Ranking Training with Large-Scale Human Feedback Datacited by 68
  8. 2021Topic-Driven and Knowledge-Aware Transformer for Dialogue Emotion Detectioncited by 56
  9. 2023PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Modelcited by 56
  10. 2022A Causal Lens for Controllable Text Generationcited by 45
  11. 2024OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flowscited by 45
  12. 2021Transformer-based Conditional Variational Autoencoder for Controllable Story Generationcited by 44
38 more
  1. 2021MuseMorphose: Full-Song and Fine-Grained Piano Music Style Transfer with One Transformer VAEcited by 39
  2. 2022DialogVED: A Pre-trained Latent Variable Encoder-Decoder Model for Dialog Response Generationcited by 35
  3. 2024Needle in the Haystack for Memory Based Large Language Modelscited by 32
  4. 2021Defending Against Backdoor Attacks in Natural Language Generationcited by 30
  5. 2022Composable Text Controls in Latent Space with ODEscited by 28
  6. 2023RECAP: Retrieval-Enhanced Context-Aware Prefix Encoder for Personalized Dialogue Response Generationcited by 25
  7. 2025Latent Thought Models with Variational Bayes Inference-Time Computationcited by 25
  8. 2021Convex Aggregation for Opinion Summarizationcited by 21
  9. 2022Fuse It More Deeply! A Variational Transformer with Layer-Wise Latent Variable Inference for Text Generationcited by 20
  10. 2023Learning to Memorize Entailment and Discourse Relations for Persona-Consistent Dialoguescited by 17
  11. 2021Finetuning Pretrained Transformers into Variational Autoencoderscited by 14
  12. 2021DiscoDVT: Generating Long Text with Discourse-Aware Discrete Variational Transformercited by 14
  13. 2022Low-Resource Dense Retrieval for Open-Domain Question Answering: A Comprehensive Surveycited by 14
  14. 2022Factual and Informative Review Generation for Explainable Recommendationcited by 14
  15. 2024Counterfactual Generation with Identifiability Guaranteescited by 13
  16. 2024Neuro-Symbolic Embedding for Short and Effective Feature Selection via Autoregressive Generationcited by 13
  17. 2022Learning Disentangled Representations for Natural Language Definitionscited by 12
  18. 2023Importance Weighted Expectation-Maximization for Protein Sequence Designcited by 12
  19. 2023MacLaSa: Multi-Aspect Controllable Text Generation via Efficient Sampling from Compact Latent Spacecited by 12
  20. 2023RegaVAE: A Retrieval-Augmented Gaussian Mixture Variational Auto-Encoder for Language Modelingcited by 12
  21. 2022An Equal-Size Hard EM Algorithm for Diverse Dialogue Generationcited by 11
  22. 2022Language Detoxification with Attribute-Discriminative Latent Spacecited by 11
  23. 2022VarMAE: Pre-training of Variational Masked Autoencoder for Domain-adaptive Language Understandingcited by 11
  24. 2022AdaVAE: Exploring Adaptive GPT-2s in Variational Auto-Encoders for Language Modelingcited by 10
  25. 2023Large-Scale and Multi-Perspective Opinion Summarization with Diverse Review Subsetscited by 10
  26. 2024Unified Generation, Reconstruction, and Representation: Generalized Diffusion with Adaptive Latent Encoding-Decodingcited by 10
  27. 2024Semformer: Transformer Language Models with Semantic Planningcited by 9
  28. 2024DiffLM: Controllable Synthetic Data Generation via Diffusion Language Modelscited by 9
  29. 2021Distantly Supervised Relation Extraction with Sentence Reconstruction and Knowledge Base Priorscited by 8
  30. 2022Prompt Conditioned VAE: Enhancing Generative Replay for Lifelong Learning in Task-Oriented Dialoguecited by 8
  31. 2021Less is More: Pre-train a Strong Text Encoder for Dense Retrieval Using a Weak Decodercited by 7
  32. 2022MulZDG: Multilingual Code-Switching Framework for Zero-shot Dialogue Generationcited by 7
  33. 2024DiVERT: Distractor Generation with Variational Errors Represented as Text for Math Multiple-choice Questionscited by 7
  34. 2024Inverse Design of Copolymers Including Stoichiometry and Chain Architecturecited by 7
  35. 2023Learning Disentangled Semantic Spaces of Explanations via Invertible Neural Networkscited by 6
  36. 2023Evaluating Open-Domain Dialogues in Latent Space with Next Sentence Prediction and Mutual Informationcited by 6
  37. 2023Graph-Induced Syntactic-Semantic Spaces in Transformer-Based Variational AutoEncoderscited by 6
  38. 2024Improving Semantic Control in Discrete Latent Spaces with Transformer Quantized Variational Autoencoderscited by 6

50 most cited of 114 All 114 (JSON)