Im2Calories: towards an automated mobile vision food diary
Abstract
We present a system which can recognize the contents
of your meal from a single image, and then predict its nutritional
contents, such as calories. The simplest version
assumes that the user is eating at a restaurant for which we
know the menu. In this case, we can collect images offline
to train a multi-label classifier. At run time, we apply the
classifier (running on your phone) to predict which foods
are present in your meal, and we lookup the corresponding
nutritional facts. We apply this method to a new dataset of
images from 23 different restaurants, using a CNN-based
classifier, significantly outperforming previous work. The
more challenging setting works outside of restaurants. In
this case, we need to estimate the size of the foods, as
well as their labels. This requires solving segmentation and
depth / volume estimation from a single image. We present
CNN-based approaches to these problems, with promising
preliminary results.
of your meal from a single image, and then predict its nutritional
contents, such as calories. The simplest version
assumes that the user is eating at a restaurant for which we
know the menu. In this case, we can collect images offline
to train a multi-label classifier. At run time, we apply the
classifier (running on your phone) to predict which foods
are present in your meal, and we lookup the corresponding
nutritional facts. We apply this method to a new dataset of
images from 23 different restaurants, using a CNN-based
classifier, significantly outperforming previous work. The
more challenging setting works outside of restaurants. In
this case, we need to estimate the size of the foods, as
well as their labels. This requires solving segmentation and
depth / volume estimation from a single image. We present
CNN-based approaches to these problems, with promising
preliminary results.
Research Areas
