mlsanityData to 7 Oct 2026

Sentence Curve Language Models

arXiv 2602.01807 ↗First posted 2 Feb 2026cs.CL cs.LGcited by 0

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Language models (LMs) are a central component of modern AI systems, and diffusion language models (DLMs) have recently emerged as a competitive alternative. Both paradigms rely on word embeddings not only to represent the input sentence, but also to represent the target sentence that backbone models are trained to predict. We argue that such static embedding of the target word is insensitive to neighboring words, encouraging locally accurate word prediction while global sentence structure is less emphasized. To address this, we propose a continuous sentence representation, termed sentence curve, defined as a spline curve whose control points affect multiple words in the sentence. Based on this representation, we introduce sentence curve language model (SCLM), which extends DLMs to predict sentence curves instead of the static word embeddings. We theoretically show that sentence curve prediction induces a regularization effect that promotes global structure modeling, and characterize how different sentence curve types affect this behavior. Empirically, SCLM achieves state-of-the-art performance among DLMs on IWSLT14 and WMT14, shows stable training without burdensome knowledge distillation, and demonstrates promising potential compared to discrete DLMs on LM1B.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 44 of its 58 references that are in the map. First posted on arXiv 2 Feb 2026.

Builds on

44 of 58 references.

Closest ancestors · 16

Same direction, newest first

  1. 2025Encoder-Decoder Diffusion Language Models for Efficient Training and Inference26 Oct
  2. 2025Soft-Masked Diffusion Language Models20 Oct
  3. 2025Unifying Continuous and Discrete Text Diffusion with Non-simultaneous Diffusion Processes28 May
  4. 2025Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models12 Mar
  5. 2025Remasking Discrete Diffusion Models with Inference-Time Scaling1 Mar
  6. 2025Large Language Diffusion Models14 Feb
  7. 2025Theoretical Benefit and Limitation of Diffusion Language Model13 Feb
  8. 2024Simple and Effective Masked Diffusion Language Models11 Jun
  9. 2023Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning23 Aug
  10. 2023AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation16 May
  11. 2023DINOISER: Diffused Conditional Sequence Learning by Manipulating Noises20 Feb
  12. 2023A Reparameterized Discrete Diffusion Model for Text Generation11 Feb
4 more
  1. 2022Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoise22 Dec
  2. 2022SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformers20 Dec
  3. 2022Empowering Diffusion Models on the Embedding Space for Text Generation19 Dec
  4. 2022Continuous diffusion for categorical data28 Nov

Other references · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Scalable Diffusion Models with Transformerscited by 5,931
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  9. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  11. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  12. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
16 more
  1. 2016Sequence-Level Knowledge Distillationcited by 761
  2. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  3. 2017Non-Autoregressive Neural Machine Translationcited by 457
  4. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  5. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  6. 2018Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinementcited by 194
  7. 2017SplineCNN: Fast Geometric Deep Learning with Continuous B-Spline Kernelscited by 160
  8. 2022Rethinking Efficient Lane Detection via Curve Modelingcited by 87
  9. 2019Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posteriorcited by 58
  10. 2019Spherical Text Embeddingcited by 49
  11. 2024From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformerscited by 34
  12. 2022On the Learning of Non-Autoregressive Transformerscited by 27
  13. 2019Understanding Composition of Word Embeddings via Tensor Decompositioncited by 4
  14. 2024N-gram Prediction and Word Difference Representations for Language Modelingcited by 3
  15. 2020Normalization of Input-output Shared Embeddings in Text Generation Modelscited by 1
  16. 2023Shared Latent Space by Both Languages in Non-Autoregressive Neural Machine Translationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Diffusion Language Models: An Experimental Analysis
  2. 2024Large Concept Models: Language Modeling in a Sentence Representation Space
  3. 2025Latent Reasoning via Sentence Embedding Prediction
  4. 2025Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective
  5. 2025A Survey on Diffusion Language Models
  6. 2025DLM-One: Diffusion Language Models for One-Step Sequence Generation
  7. 2026Scaling and Distilling Text Embeddings for Better Diffusibility
  8. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Models