Google Research

Semantic MapNet: Building Allocentric Semantic Maps and Representations from Egocentric Views

  • Vincent Cartillier
  • Zhile Ren
  • Neha Jain
  • Stefan Lee
  • Irfan Essa
  • Dhruv Batra
in Proceedings of AAAI 2021, AAAI

Abstract

We study the task of semantic mapping – specifically, an embodied agent (a robot or an egocentric AI assistant) is given a tour of a new environment and asked to build an allocentric top-down semantic map (‘what is where?’) from egocentric observations of an RGB-D camera with known pose (via localization sensors). Towards this goal, we present Semantic

MapNet (SMNet), which consists of: (1) an Egocentric Visual Encoder that encodes each egocentric RGB-D frame, (2) a Feature Projector that projects egocentric features to appropriate locations on a floor-plan, (3) a Spatial Memory Tensor of size floor-plan length × width × feature-dims that learns to accumulate projected egocentric features, and (4) a Map Decoder that uses the memory tensor to produce semantic top-down maps. SMNet combines the strengths of (known) projective camera geometry and neural representation learning.

On the task of semantic mapping in the Matterport3D dataset, SMNet significantly outperforms competitive baselines by 4.01 − 16.81% (absolute) on mean-IoU and 3.81 − 19.69% (absolute) on Boundary-F1 metrics. Moreover, we show how to use the neural episodic memories and spatio-semantic allocentric representations built by SMNet for subsequent tasks in the same space – navigating to objects seen during the tour (‘Find chair’) or answering questions about the space (‘How many chairs did you see in the house?’).

Project page:https://vincentcartillier.github.io/smnet.html. Also available as arXiv preprint arXiv:2010.01191 (https://arxiv.org/abs/2010.01191)

Learn more about how we do research

We maintain a portfolio of research projects, providing individuals and teams the freedom to emphasize specific types of work