OVI-MAP: Open-Vocabulary Instance-Semantic Mapping

Incremental open-set 3D semantic segmentation in real-world environments is an essential task for autonomous systems, enabling them to perceive and understand their surroundings in an open-world setting. Segmenting the instance and obtaining their semantic information while exploring in the scene is a challenging task due to the complexity of 3D scene and the limitations of current Vision-Language Models (VLMs) in extracting semantic features. The need for consistent long-term segmentation and real-time processing further complicates the problem. This paper presents a novel approach that achieves incremental open-set instance recognition in the scene, enabling efficient scene exploration. Our method combines instance map reconstruction and semantic extraction with VLMs to perform incremental 3D semantic segmentation in an open-set manner. The proposed method outperforms the state-of-the-art methods in terms of both segmentation accuracy and computation efficiency, demonstrating its effectiveness in real-world applications. The code would be made public.