mlsanityData to 7 Oct 2026

A Mathematical Abstraction for Balancing the Trade-off Between Creativity and Reality in Large Language Models

arXiv 2306.02295 ↗First posted 4 Jun 2023cs.CL cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

Large Language Models have become popular for their remarkable capabilities in human-oriented tasks and traditional natural language processing tasks. Its efficient functioning is attributed to the attention mechanism in the Transformer architecture, enabling it to concentrate on particular aspects of the input. LLMs are increasingly being used in domains such as generating prose, poetry or art, which require the model to be creative (e.g. Adobe firefly). LLMs possess advanced language generation abilities that enable them to generate distinctive and captivating content. This utilization of LLMs in generating narratives shows their flexibility and potential for use in domains that extend beyond conventional natural language processing duties. In different contexts, we may expect the LLM to generate factually correct answers, that match reality; e.g., question-answering systems or online assistants. In such situations, being correct is critical to LLMs being trusted in practice. The Bing Chatbot provides its users with the flexibility to select one of the three output modes: creative, balanced, and precise. Each mode emphasizes creativity and factual accuracy differently. In this work, we provide a mathematical abstraction to describe creativity and reality based on certain losses. A model trained on these losses balances the trade-off between the creativity and reality of the model.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 67 of its 89 references that are in the map. First posted on arXiv 4 Jun 2023.

Builds on

67 of 89 references.

References · 67

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  10. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  11. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  12. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
38 more
  1. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  2. 2018Universal Transformerscited by 613
  3. 2019Designing and Interpreting Probes with Control Taskscited by 608
  4. 2019Are Transformers universal approximators of sequence-to-sequence functions?cited by 404
  5. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  6. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  7. 2017Train longer, generalize better: closing the generalization gap in large batch training of neural networkscited by 278
  8. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  9. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
  10. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  11. 2021Scatterbrain: Unifying Sparse and Low-rank Attention Approximationcited by 128
  12. 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
  13. 2021Self-Attention Networks Can Process Bounded Hierarchical Languagescited by 105
  14. 2022Unveiling Transformers with LEGO: a synthetic reasoning taskcited by 105
  15. 2023Fast Attention Requires Bounded Entriescited by 102
  16. 2021Statistically Meaningful Approximation: a Case Study on Approximating Turing Machines with Transformerscited by 101
  17. 2018Solving Linear Programs in the Current Matrix Multiplication Timecited by 100
  18. 2017CAN: Creative Adversarial Networks, Generating "Art" by Learning About Styles and Deviating from Style Normscited by 96
  19. 2020On the Computational Power of Transformers and its Implications in Sequence Modelingcited by 85
  20. 2022A very preliminary analysis of DALL-E 2cited by 82
  21. 2019Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networkscited by 76
  22. 2021Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Modelscited by 75
  23. 2021Training Multi-Layer Over-Parametrized Neural Network in Subquadratic Timecited by 62
  24. 2020Training (Overparametrized) Neural Networks in Near-Linear Timecited by 58
  25. 2023KDEformer: Accelerating Transformers via Kernel Density Estimationcited by 58
  26. 2020An Improved Cutting Plane Method for Convex Optimization, Convex-Concave Games and its Applicationscited by 57
  27. 2020A Refined Laser Method and Faster Matrix Multiplicationcited by 56
  28. 2019Solving Empirical Risk Minimization in the Current Matrix Multiplication Timecited by 53
  29. 2020A Faster Interior Point Method for Semidefinite Programmingcited by 53
  30. 2020How Can Self-Attention Networks Recognize Dyck-n Languages?cited by 51
  31. 2023Attention Scheme Inspired Softmax Regressioncited by 49
  32. 2023An Over-parameterized Exponential Regressioncited by 45
  33. 2020Faster Dynamic Matrix Inverse for Faster LPscited by 44
  34. 2023Solving Regularized Exp, Cosh and Sinh Regression Problemscited by 44
  35. 2020A Mathematical Theory of Attentioncited by 43
  36. 2019A Deterministic Linear Program Solver in Current Matrix Multiplication Timecited by 42
  37. 2023The Closeness of In-Context Learning and Weight Shifting for Softmax Regressioncited by 41
  38. 2019Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problemscited by 40

50 most cited of 67 All 67 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 18; 2024: 2; 2025: 8; 2026: 1.

29 papers.

Sort:

By area:

  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
  2. 2023In-Context Learning for Attention Scheme: from Single Softmax Regression to Multiple Softmax Regression via a Tensor Trickcited by 32
  3. 2024Conv-Basis: A New Paradigm for Efficient Attention Inference and Gradient Computation in Transformerscited by 32
  4. 2023A Fast Optimization View: Reformulating Single Layer Attention in LLM Based on Tensor and SVM Trick, and Solving It in Matrix Multiplication Timecited by 30
  5. 2023InfoPrompt: Information-Theoretic Soft Prompt Tuning for Natural Language Understandingcited by 29
  6. 2023GradientCoin: A Peer-to-Peer Decentralized Large Language Modelscited by 23
  7. 2024The Computational Limits of State-Space Models and Mamba via the Lens of Circuit Complexitycited by 21
  8. 2023Solving Attention Kernel Regression Problem via Pre-conditionercited by 20
  9. 2023Beyond Classical Attention: Quantum Attention for Scalable Computationcited by 18
  10. 2023Zero-th Order Algorithm for Softmax Attention Optimizationcited by 16
  11. 2023A Unified Scheme of ResNet and Softmaxcited by 12
  12. 2023Efficient SGD Neural Network Training via Sublinear Activated Neuron Identificationcited by 11
17 more
  1. 2023Unmasking Transformers: A Theoretical Approach to Data Recovery via Attention Weightscited by 11
  2. 2023Efficient Alternating Minimization with Applications to Weighted Low Rank Approximationcited by 9
  3. 2025Theoretical Guarantees for High Order Trajectory Refinement in Generative Flowscited by 9
  4. 2023Is Solving Graph Neural Tangent Kernel Equivalent to Training Graph Neural Network?cited by 8
  5. 2023The Expressibility of Polynomial based Attention Schemecited by 7
  6. 2023A Theoretical Insight into Attack and Defense of Gradient Leakage in Transformercited by 6
  7. 2023One Pass Streaming Algorithm for Super Long Token Attention Approximation in Sublinear Spacecited by 6
  8. 2025Breaking the Frozen Subspace: Importance Sampling for Low-Rank Optimization in LLM Pretrainingcited by 6
  9. 2025Has the Creativity of Large-Language Models peaked? An analysis of inter- and intra-LLM variabilitycited by 6
  10. 2025Scaling Law Phenomena Across Regression Paradigms: Multiple and Kernel Approachescited by 4
  11. 2025Neural Algorithmic Reasoning for Hypergraphs with Looped Transformerscited by 3
  12. 2025Time and Memory Trade-off of KV-Cache Compression in Tensor Transformer Decodingcited by 2
  13. 2025Theoretical Foundation of Flow-Based Time Series Generation: Provable Approximation, Generalization, and Efficiencycited by 2
  14. 2025From Hallucination to Reliability: Generative Modeling and the Structure of Scientific Inferencecited by 2
  15. 2026Power and Limitations of Aggregation in Compound AI Systemscited by 1
  16. 2023Online Adaptive Mahalanobis Distance Estimationcited by 0
  17. 2023Fast Heavy Inner Product Identification Between Weights and Inputs in Neural Network Trainingcited by 0