Object-X: Learning to Reconstruct Multi-Modal 3D Object Representations

3D scene representation is a crucial aspect of computer vision, enabling applications such as robotics and augmented reality. Traditional methods like point clouds require significant storage, hindering their scalability and practicality. While 3D scene graphs offer a lightweight alternative by representing scenes as interconnected objects, they often lack the geometric and visual detail needed for many tasks. In this work, we introduce a novel method Graph2Splat that learns rich scene graph node embeddings, enabling both efficient 3D scene reconstruction and direct utilization of the embeddings for other downstream tasks. Our method predicts a 3D Gaussian Splat (3DGS) representation for each object from its corresponding embedding, effectively capturing both appearance and geometry. Evaluated on two real-world datasets, our approach achieves high-fidelity novel view synthesis comparable to direct 3DGS reconstruction, while significantly improving geometric accuracy. Importantly, this is achieved with three orders of magnitude less storage than standard approaches, making it a practical and versatile solution for 3D scene representation.
The code and models will be made public.