# Embedding-Aligned Language Models

arXiv: https://arxiv.org/abs/2406.00024 · HTML: https://mlsanity.com/paper/2406.00024/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 39 of its 55 references that are in the map.
- First posted on arXiv 24 May 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 30 of 39)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [Automatic chemical design using a data-driven continuous representation of molecules](https://mlsanity.com/paper/1610.02415/) (2016)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback](https://mlsanity.com/paper/2309.00267/) (2023)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2305.11747/) (2023)
- [Chain-of-Verification Reduces Hallucination in Large Language Models](https://mlsanity.com/paper/2309.11495/) (2023)
- [Variational Autoencoders for Collaborative Filtering](https://mlsanity.com/paper/1802.05814/) (2018)
- [Detecting and Preventing Hallucinations in Large Vision Language Models](https://mlsanity.com/paper/2308.06394/) (2023)
- [Accelerating Reinforcement Learning with Learned Skill Priors](https://mlsanity.com/paper/2010.11944/) (2020)
- [MineRL: A Large-Scale Dataset of Minecraft Demonstrations](https://mlsanity.com/paper/1907.13440/) (2019)
- [Fine-tuning language models to find agreement among humans with diverse preferences](https://mlsanity.com/paper/2211.15006/) (2022)
- [Latent Space Oddity: on the Curvature of Deep Generative Models](https://mlsanity.com/paper/1710.11379/) (2017)
- [Improved Variational Autoencoders for Text Modeling using Dilated Convolutions](https://mlsanity.com/paper/1702.08139/) (2017)
- [Action Space Shaping in Deep Reinforcement Learning](https://mlsanity.com/paper/2004.00980/) (2020)
- [Geometrically Enriched Latent Spaces](https://mlsanity.com/paper/2008.00565/) (2020)
- [Contextual Bandits with Large Action Spaces: Made Practical](https://mlsanity.com/paper/2207.05836/) (2022)
- [The Natural Language of Actions](https://mlsanity.com/paper/1902.01119/) (2019)
- [Aligning Language Models with Human Preferences via a Bayesian Approach](https://mlsanity.com/paper/2310.05782/) (2023)

## Built on by (most cited first, 9 of 9)

- [Rethinking Diverse Human Preference Learning through Principal Component Analysis](https://mlsanity.com/paper/2502.13131/) (2025)
- [Preference Adaptive and Sequential Text-to-Image Generation](https://mlsanity.com/paper/2412.10419/) (2024)
- [Comparing Few to Rank Many: Active Human Preference Learning using Randomized Frank-Wolfe](https://mlsanity.com/paper/2412.19396/) (2024)
- [Incentive-Aware Multi-Fidelity Optimization for Generative Advertising in Large Language Models](https://mlsanity.com/paper/2604.06263/) (2026)
- [Descriptive History Representations: Learning Representations by Answering Questions](https://mlsanity.com/paper/2506.02125/) (2025)
- [FuDoBa: Fusing Document and Knowledge Graph-based Representations with Bayesian Optimisation](https://mlsanity.com/paper/2507.06622/) (2025)
- [RLRF: Competitive Search Agent Design via Reinforcement Learning from Ranker Feedback](https://mlsanity.com/paper/2510.04096/) (2025)
- [Do Reasoning Models Enhance Embedding Models?](https://mlsanity.com/paper/2601.21192/) (2026)
- [OLLM: Options-based Large Language Models](https://mlsanity.com/paper/2604.19087/) (2026)

## Abstract (arXiv)

We propose a novel approach for training large language models (LLMs) to adhere to objectives defined within a latent embedding space. Our method leverages reinforcement learning (RL), treating a pre-trained LLM as an environment. Our embedding-aligned guided language (EAGLE) agent is trained to iteratively steer the LLM's generation towards optimal regions of the latent embedding space, w.r.t. some predefined criterion. We demonstrate the effectiveness of the EAGLE agent using the MovieLens 25M and Amazon Review datasets to surface content gaps that satisfy latent user demand. We also demonstrate the benefit of using an optimal design of a state-dependent action set to improve EAGLE's efficiency. Our work paves the way for controlled and grounded text generation using LLMs, ensuring consistency with domain-specific knowledge and data representations.