mlsanityData to 7 Oct 2026

TOSS:High-quality Text-guided Novel View Synthesis from a Single Image

arXiv 2310.10644 ↗First posted 16 Oct 2023cs.CVcited by 26

Not on the map. Why?

Abstract (arXiv)

In this paper, we present TOSS, which introduces text to the task of novel view synthesis (NVS) from just a single RGB image. While Zero-1-to-3 has demonstrated impressive zero-shot open-set NVS capability, it treats NVS as a pure image-to-image translation problem. This approach suffers from the challengingly under-constrained nature of single-view NVS: the process lacks means of explicit user control and often results in implausible NVS generations. To address this limitation, TOSS uses text as high-level semantic information to constrain the NVS solution space. TOSS fine-tunes text-to-image Stable Diffusion pre-trained on large-scale text-image pairs and introduces modules specifically tailored to image and camera pose conditioning, as well as dedicated training for pose correctness and preservation of fine details. Comprehensive experiments are conducted with results showing that our proposed TOSS outperforms Zero-1-to-3 with more plausible, controllable and multiview-consistent NVS results. We further support these results with comprehensive ablations that underscore the effectiveness and potential of the introduced semantic guidance and architecture design.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 54 of its 60 references that are in the map. First posted on arXiv 16 Oct 2023.

Builds on

54 of 60 references.

References · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2022Generative Adversarial Networkscited by 5,603
  11. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  12. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
38 more
  1. 2022Instant Neural Graphics Primitives with a Multiresolution Hash Encodingcited by 3,698
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  3. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  4. 2019DeepSDF: Learning Continuous Signed Distance Functions for Shape Representationcited by 2,682
  5. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  6. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  7. 2020pixelNeRF: Neural Radiance Fields from One or Few Imagescited by 1,574
  8. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  9. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,522
  10. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  11. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  12. 2022Magic3D: High-Resolution Text-to-3D Content Creationcited by 1,285
  13. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  14. 2021Efficient Geometry-aware 3D Generative Adversarial Networkscited by 1,204
  15. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  16. 2022Diffusion Models in Vision: A Surveycited by 1,008
  17. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  18. 2023GLIGEN: Open-Set Grounded Text-to-Image Generationcited by 774
  19. 2023SyncDreamer: Generating Multiview-consistent Images from a Single-view Imagecited by 638
  20. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  21. 2022Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Itemscited by 601
  22. 2023One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimizationcited by 555
  23. 2023Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priorscited by 395
  24. 2023Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Priorcited by 341
  25. 2023RealFusion: 360deg Reconstruction of Any Object from a Single Imagecited by 305
  26. 2022Perception Prioritized Training of Diffusion Modelscited by 301
  27. 2022Novel View Synthesis with Diffusion Modelscited by 299
  28. 2021StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generationcited by 289
  29. 2022SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generationcited by 287
  30. 2023Generative Novel View Synthesis with 3D-Aware Diffusion Modelscited by 273
  31. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  32. 2023MeshDiffusion: Score-based Generative 3D Mesh Modelingcited by 198
  33. 20233DGen: Triplane Latent Diffusion for Textured Mesh Generationcited by 181
  34. 2022NeRDi: Single-View NeRF Synthesis with Language-Guided Diffusion as General Image Priorscited by 156
  35. 2023Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3D Generationcited by 133
  36. 2023Locally Attentional SDF Diffusion for Controllable 3D Shape Generationcited by 129
  37. 2023HumanSD: A Native Skeleton-Guided Diffusion Model for Human Image Generationcited by 104
  38. 2023DreamWaltz: Make a Scene with Complex 3D Animatable Avatarscited by 82

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 13; 2025: 9.

26 papers.

Sort:

By area:

  1. 2024SV4D: Dynamic 3D Content Generation with Multi-Frame and Multi-View Consistencycited by 143
  2. 2024MV-Adapter: Multi-view Consistent Image Generation Made Easycited by 111
  3. 2024MotionLLM: Understanding Human Behaviors from Human Motions and Videoscited by 89
  4. 2024REPARO: Compositional 3D Assets Generation with Differentiable 3D Layout Alignmentcited by 45
  5. 2025SV4D 2.0: Enhancing Spatio-Temporal Consistency in Multi-View Video Diffusion for High-Quality 4D Generationcited by 44
  6. 2023Cascade-Zero123: One Image to Highly Consistent 3D with Self-Prompted Nearby Viewscited by 32
  7. 2023Stable Score Distillation for High-Quality 3D Generationcited by 27
  8. 2023HarmonyView: Harmonizing Consistency and Diversity in One-Image-to-3Dcited by 27
  9. 2024GaussianDreamerPro: Text to Manipulable 3D Gaussians with Highly Enhanced Qualitycited by 17
  10. 2025Simulating the Real World: A Unified Survey of Multimodal Generative Modelscited by 14
  11. 2024A Survey on 3D Human Avatar Modeling -- From Reconstruction to Generationcited by 13
  12. 2024FlexiDreamer: Single Image-to-3D Generation with FlexiCubescited by 12
14 more
  1. 2024LiftImage3D: Lifting Any Single Image to 3D Gaussians with Video Generation Priorscited by 11
  2. 2025UP2You: Fast Reconstruction of Yourself from Unconstrained Photo Collectionscited by 10
  3. 2025CharacterShot: Controllable and Consistent 4D Character Animationcited by 9
  4. 2024FlexGen: Flexible Multi-View Generation from Text and Image Inputscited by 8
  5. 2024GECO: Generative Image-to-3D within a SECOndcited by 6
  6. 2024X-Diffusion: Generating Detailed 3D MRI Volumes From a Single Image Using Cross-Sectional Diffusion Modelscited by 5
  7. 2025Stable Part Diffusion 4D: Multi-View RGB and Kinematic Parts Video Generationcited by 5
  8. 2025FB-4D: Spatial-Temporal Coherent Dynamic 3D Content Generation with Feature Bankscited by 3
  9. 2024Ctrl123: Consistent Novel View Synthesis via Closed-Loop Transcriptioncited by 2
  10. 2023HawkI: Homography & Mutual Information Guidance for 3D-free Single Image to Aerial Viewcited by 1
  11. 2025DreamComposer++: Empowering Diffusion Models with Multi-View Conditions for 3D Content Generationcited by 1
  12. 2025UniView: Enhancing Novel View Synthesis From A Single Image By Unifying Reference Featurescited by 1
  13. 20243D-free meets 3D priors: Novel View Synthesis from a Single Image with Pretrained Diffusion Guidancecited by 0
  14. 2025Dream3DAvatar: Text-Controlled 3D Avatar Reconstruction from a Single Imagecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Novel View Synthesis with Pixel-Space Diffusion Models
  2. 2026All-in-One Conditioning for Text-to-Image Synthesis
  3. 2026SemanticNVS: Improving Semantic Scene Understanding in Generative Novel View Synthesis
  4. 2024FSViewFusion: Few-Shots View Generation of Novel Objects
  5. 2024NVComposer: Boosting Generative Novel View Synthesis with Multiple Sparse and Unposed Images
  6. 2023PanGu-Draw: Advancing Resource-Efficient Text-to-Image Synthesis with Time-Decoupled Training and Reusable Coop-Diffusion
  7. 2024From Text to Pose to Image: Improving Diffusion Model Control and Quality
  8. 2024Training-free Composite Scene Generation for Layout-to-Image Synthesis