Discovering User Bias in Ordinal Voting Systems
Abstract
Crowdsourcing systems increasingly rely on users to provide more
subjective ground truth for intelligent systems - e.g. ratings, aspect
of quality and perspectives on how expensive or lively a place feels,
etc. We focus on the ubiquitous implementation of online user ordinal voting (e.g 1-5, 1 star-4 stars) on some aspect of an entity, to
extract a relative truth, measured by a selected metric such as vote
plurality or mean. We argue that this methodology can aggregate
results that yield little information to the end user. In particular,
ordinal user rankings often converge to a indistinguishable rating.
This is demonstrated by the trend in certain cities for the majority of restaurants to all have a 4 star rating. Similarly, the rating of an establishment can be significantly affected by a few users.
User bias in voting is not spam, but rather a preference that can
be harnessed to provide more information to users. We explore
notions of both global skew and user bias. Leveraging these bias
and preference concepts, the paper suggests explicit models for
better personalization and more informative ratings.
subjective ground truth for intelligent systems - e.g. ratings, aspect
of quality and perspectives on how expensive or lively a place feels,
etc. We focus on the ubiquitous implementation of online user ordinal voting (e.g 1-5, 1 star-4 stars) on some aspect of an entity, to
extract a relative truth, measured by a selected metric such as vote
plurality or mean. We argue that this methodology can aggregate
results that yield little information to the end user. In particular,
ordinal user rankings often converge to a indistinguishable rating.
This is demonstrated by the trend in certain cities for the majority of restaurants to all have a 4 star rating. Similarly, the rating of an establishment can be significantly affected by a few users.
User bias in voting is not spam, but rather a preference that can
be harnessed to provide more information to users. We explore
notions of both global skew and user bias. Leveraging these bias
and preference concepts, the paper suggests explicit models for
better personalization and more informative ratings.
Research Areas
