Benchmarking Intelligent Service Robots through Scientific Competitions: the RoboCup@Home approach

The dynamical and uncertain environments of domestic service robots, which include humans, require rethinking of the benchmarking principles for testing these robots. Since 2006 RoboCup@Home has used statistical procedures to track and steer the progress of domestic service robots. This paper explains the procedures and shows outcomes of these international benchmarking efforts. Although aspects such as shopping in a supermarket receive a fair amount of attention in the robotics community, the authors believe that a recently implemented test is the most important outcome of RoboCup@Home, namely the benchmarking of robot cognition.