mlsanityData to 14 Oct 2026

SceneScore: Learning a Cost Function for Object Arrangement

arXiv 2311.08530 ↗First posted 14 Nov 2023cs.RO cs.CV cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Arranging objects correctly is a key capability for robots which unlocks a wide range of useful tasks. A prerequisite for creating successful arrangements is the ability to evaluate the desirability of a given arrangement. Our method "SceneScore" learns a cost function for arrangements, such that desirable, human-like arrangements have a low cost. We learn the distribution of training arrangements offline using an energy-based model, solely from example images without requiring environment interaction or human supervision. Our model is represented by a graph neural network which learns object-object relations, using graphs constructed from images. Experiments demonstrate that the learned cost function can be used to predict poses for missing objects, generalise to novel objects using semantic features, and can be composed with other cost functions to satisfy constraints at inference time.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 29 of its 39 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

29 of 39 references.

References · 29

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2017Mask R-CNNcited by 7,277
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  6. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  7. 2021Implicit Behavioral Cloningcited by 452
  8. 2019Sliced Score Matching: A Scalable Approach to Density and Score Estimationcited by 384
  9. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 344
  10. 2021How to Train Your Energy-Based Modelscited by 274
  11. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 221
  12. 2016A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Modelscited by 168
17 more
  1. 2020Rearrangement: A Challenge for Embodied AIcited by 155
  2. 2023Task and Motion Planning with Large Language Models for Object Rearrangementcited by 141
  3. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Roboticscited by 106
  4. 2023LEGO-Net: Learning Regular Rearrangements of Objects in Roomscited by 76
  5. 20196-DOF Grasping for Target-driven Object Manipulation in Cluttercited by 70
  6. 2021StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objectscited by 59
  7. 2022Housekeep: Tidying Virtual Households using Commonsense Reasoningcited by 57
  8. 2022StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objectscited by 50
  9. 2022IFOR: Iterative Flow Minimization for Robotic Object Rearrangementcited by 42
  10. 2022TIDEE: Tidying Up Novel Rooms using Visuo-Semantic Commonsense Priorscited by 30
  11. 2023Energy-based Models are Zero-Shot Planners for Compositional Scene Rearrangementcited by 30
  12. 2021My House, My Rules: Learning Tidying Preferences with Graph Neural Networkscited by 27
  13. 2021Efficient and Interpretable Robot Manipulation with Graph Neural Networkscited by 22
  14. 2023CabiNet: Scaling Neural Collision Detection for Object Rearrangement with Procedural Scene Generationcited by 18
  15. 2022Implicit Kinematic Policies: Unifying Joint and Cartesian Action Spaces in End-to-End Robot Learningcited by 15
  16. 2022Transformers are Adaptable Task Plannerscited by 15
  17. 2018Concept Learning with Energy-Based Modelscited by 14

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 2.

4 papers.

  1. 2023Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Modelscited by 16
  2. 2024Geometric Algebra Meets Large Language Models: Instruction-Based Transformations of Separate Meshes in 3D, Interactive and Controllable Scenescited by 1
  3. 2025Tidiness Score-Guided Monte Carlo Tree Search for Visual Tabletop Rearrangementcited by 1
  4. 2025Explaining Why Things Go Where They Go: Interpretable Constructs of Human Organizational Preferencescited by 0