Quizz: Targeted Crowdsourcing with a Billion (Potential) Users
Abstract
We describe Quizz, a gamified crowdsourcing system that simultaneously assesses
the knowledge of users and acquires new knowledge from them. Quizz operates by
asking users to complete short quizzes on specific topics; as a user answers
the quiz questions, Quizz estimates the user's competence. To acquire new
knowledge, Quizz also incorporates questions for which we do not have a known
answer; the answers given by competent users provide useful signals for
selecting the correct answers for these questions. Quizz actively tries to
identify knowledgeable users on the Internet by running advertising campaigns,
effectively leveraging the targeting capabilities of existing,
publicly available, ad placement services. Quizz quantifies the contributions
of the users using information theory and sends feedback to the advertising
system about each user. The feedback allows the ad targeting mechanism to
further optimize ad placement.
Our experiments, which involve over ten thousand users, confirm that we can
crowdsource knowledge curation for niche and specialized topics, as the
advertising network can automatically identify users with the desired expertise
and interest in the given topic. We present controlled experiments that examine
the effect of various incentive mechanisms, highlighting the need for having
short-term rewards as goals, which incentivize the users to contribute.
Finally, our cost-quality analysis indicates that the cost of our approach is
below that of hiring workers through paid-crowdsourcing platforms, while
offering the additional advantage of giving access to billions of potential
users all over the planet, and being able to reach users with specialized
expertise that is not typically available through existing labor marketplaces.
the knowledge of users and acquires new knowledge from them. Quizz operates by
asking users to complete short quizzes on specific topics; as a user answers
the quiz questions, Quizz estimates the user's competence. To acquire new
knowledge, Quizz also incorporates questions for which we do not have a known
answer; the answers given by competent users provide useful signals for
selecting the correct answers for these questions. Quizz actively tries to
identify knowledgeable users on the Internet by running advertising campaigns,
effectively leveraging the targeting capabilities of existing,
publicly available, ad placement services. Quizz quantifies the contributions
of the users using information theory and sends feedback to the advertising
system about each user. The feedback allows the ad targeting mechanism to
further optimize ad placement.
Our experiments, which involve over ten thousand users, confirm that we can
crowdsource knowledge curation for niche and specialized topics, as the
advertising network can automatically identify users with the desired expertise
and interest in the given topic. We present controlled experiments that examine
the effect of various incentive mechanisms, highlighting the need for having
short-term rewards as goals, which incentivize the users to contribute.
Finally, our cost-quality analysis indicates that the cost of our approach is
below that of hiring workers through paid-crowdsourcing platforms, while
offering the additional advantage of giving access to billions of potential
users all over the planet, and being able to reach users with specialized
expertise that is not typically available through existing labor marketplaces.
Research Areas
