mlsanityData to 7 Oct 2026

LlaMaVAE: Guiding Large Language Model Generation via Continuous Latent Sentence Spaces

arXiv 2312.13208 ↗First posted 20 Dec 2023cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

Deep generative neural networks, such as Variational AutoEncoders (VAEs), offer an opportunity to better understand and control language models from the perspective of sentence-level latent spaces. To combine the controllability of VAE latent spaces with the state-of-the-art performance of recent large language models (LLMs), we present in this work LlaMaVAE, which combines expressive encoder and decoder models (sentenceT5 and LlaMA) with a VAE architecture, aiming to provide better text generation control to LLMs. In addition, to conditionally guide the VAE generation, we investigate a new approach based on flow-based invertible neural networks (INNs) named Invertible CVAE. Experimental results reveal that LlaMaVAE can outperform the previous state-of-the-art VAE language model, Optimus, across various tasks, including language modelling, semantic textual similarity and definition modelling. Qualitative analysis on interpolation and traversal experiments also indicates an increased degree of semantic clustering and geometric consistency, which enables better generation control.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 42 of its 58 references that are in the map. First posted on arXiv 20 Dec 2023.

Builds on

42 of 58 references.

References · 42

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  3. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  4. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  5. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  6. 2016Density estimation using Real NVPcited by 2,029
  7. 2019Language Models as Knowledge Bases?cited by 1,922
  8. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  9. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  10. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  11. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  12. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
30 more
  1. 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
  2. 2015Generating Sentences from a Continuous Spacecited by 607
  3. 2021Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Modelscited by 462
  4. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
  5. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  6. 2015Adversarial Autoencoderscited by 359
  7. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  8. 2019Residual Flows for Invertible Generative Modelingcited by 256
  9. 2017Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoderscited by 222
  10. 2019Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishingcited by 191
  11. 2018Neural Importance Samplingcited by 154
  12. 2018Disentangled Representation Learning for Non-Parallel Text Style Transfercited by 127
  13. 2020X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Modelscited by 117
  14. 2020Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Spacecited by 114
  15. 2018WorldTree: A Corpus of Explanation Graphs for Elementary Science Questions supporting Multi-Hop Inferencecited by 60
  16. 2019Generating Sentences from Disentangled Syntactic and Semantic Spacescited by 59
  17. 2023normflows: A PyTorch Package for Normalizing Flowscited by 56
  18. 2020Pre-Training Transformers as Energy-Based Cloze Modelscited by 52
  19. 2016Definition Modeling: Learning to define word embeddings in natural languagecited by 46
  20. 2021Transformer-based Conditional Variational Autoencoder for Controllable Story Generationcited by 44
  21. 2020Differentiable Reasoning over a Virtual Knowledge Basecited by 39
  22. 2019A Surprisingly Effective Fix for Deep Latent Variable Modeling of Textcited by 32
  23. 2018On the Latent Space of Wasserstein Auto-Encoderscited by 22
  24. 2022Semeval-2022 Task 1: CODWOE -- Comparing Dictionaries and Word Embeddingscited by 18
  25. 2021Hybrid Autoregressive Inference for Scalable Multi-hop Explanation Regenerationcited by 17
  26. 2022Learning Disentangled Representations for Natural Language Definitionscited by 12
  27. 2023Learning Disentangled Semantic Spaces of Explanations via Invertible Neural Networkscited by 6
  28. 2023Graph-Induced Syntactic-Semantic Spaces in Transformer-Based Variational AutoEncoderscited by 6
  29. 2022Quasi-symbolic Semantic Geometry over Transformer-based Variational AutoEncodercited by 5
  30. 2019Two Birds with One Stone: Investigating Invertible Neural Networks for Inverse Problems in Morphologycited by 3

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2024Improving Semantic Control in Discrete Latent Spaces with Transformer Quantized Variational Autoencoderscited by 6
  2. 2025LangVAE and LangSpace: Building and Probing for Language Model VAEscited by 3
  3. 2025Bridging Compositional and Distributional Semantics: A Survey on Latent Semantic Geometry via AutoEncodercited by 0