Martin Pielot
Martin Pielot is currently working as Quantitative User Experience Researcher (Quant UXR) at Google, ensuring that Google’s Material Design System helps designers and developers to create beautiful and usable user interfaces and experiences. In the past, he worked in the spaces of notifications & interruptions and tactile user interfaces. His publications received over 4000 citations and several best paper awards and honorable mentions.
Research Areas
Authored Publications
Sort By
Did You Misclick? Reversing 5-Point Satisfaction Scales Causes Unintended Responses
Mario Callegaro
CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems
Preview abstract
When fielding satisfaction questions, survey platforms offer the option to randomly reverse the response options. In this paper, we provide evidence that the use of this option leads to biased results. In Study 1, we show that reversing vertically oriented response options leads to significantly lower satisfaction ratings – from 90 to 82 percent in our case. Study 2 had survey respondents verify their response and found that on a reversed scale, the very-dissatisfied option was selected unintentionally in about half of the cases. The cause, shown by Study 3, is that survey respondents expect the positive option at the top and do not always pay sufficient attention to the question, combined with the similar spelling of satisfied and dissatisfied. To prevent unintentional responses from biasing the results, we recommend keeping the positive option at the top in vertically-oriented scales with visually-similar endpoint labels.
View details
Effects on reversing a 5 point vertically oriented satisfaction question, results from 3 studies using bipolar and unipolar satisfaction scales
Mario Callegaro
European Survey Research Association 2023 Conference University of Milano Bicocca
Preview abstract
Customer satisfaction surveys are common in technology companies like Google. The standard satisfaction question asks respondents to rate how satisfied or dissatisfied they are with a product or service generally going from very satisfied to very dissatisfied.
When the scale is presented vertically, some survey literature suggests placing the positive end of the scale on top as “up means good” to avoid confusing respondents.
We report from 2 studies. The first study shows that reversing the response options of a bipolar satisfaction question (very dissatisfied on top) leads to significantly lower reported satisfaction. In a between group experiment, 3,000 Google Opinion Rewards (Smartphone panel) respondents took a 1-question satisfaction survey. When the response options were reversed participants were 10 times more likely to select the very dissatisfied option (5% versus 0.5% prevalence). They also took 11% more time to answer the reversed scale.
The second study shows that this effect can be partially explained by respondents mistaking the word dissatisfied for satisfied. ~1750 people responded to a reversed satisfaction question in an in-product survey on fonts.google.com. In a follow-up verification question (“You selected [answer option], was this your intention?”), 42.1% of the respondents indicated that they had selected very dissatisfied by mistake. Open ended feedback suggests that respondents hadn’t read properly and expected the positive option on top.
More experiments should be conducted on different samples to better understand the interaction of scale orientation versus the type of scale (unipolar vs. bipolar).
View details
