Effects on reversing a 5 point vertically oriented satisfaction question, results from 3 studies using bipolar and unipolar satisfaction scales
Abstract
Customer satisfaction surveys are common in technology companies like Google. The standard satisfaction question asks respondents to rate how satisfied or dissatisfied they are with a product or service generally going from very satisfied to very dissatisfied.
When the scale is presented vertically, some survey literature suggests placing the positive end of the scale on top as “up means good” to avoid confusing respondents.
We report from 2 studies. The first study shows that reversing the response options of a bipolar satisfaction question (very dissatisfied on top) leads to significantly lower reported satisfaction. In a between group experiment, 3,000 Google Opinion Rewards (Smartphone panel) respondents took a 1-question satisfaction survey. When the response options were reversed participants were 10 times more likely to select the very dissatisfied option (5% versus 0.5% prevalence). They also took 11% more time to answer the reversed scale.
The second study shows that this effect can be partially explained by respondents mistaking the word dissatisfied for satisfied. ~1750 people responded to a reversed satisfaction question in an in-product survey on fonts.google.com. In a follow-up verification question (“You selected [answer option], was this your intention?”), 42.1% of the respondents indicated that they had selected very dissatisfied by mistake. Open ended feedback suggests that respondents hadn’t read properly and expected the positive option on top.
More experiments should be conducted on different samples to better understand the interaction of scale orientation versus the type of scale (unipolar vs. bipolar).
When the scale is presented vertically, some survey literature suggests placing the positive end of the scale on top as “up means good” to avoid confusing respondents.
We report from 2 studies. The first study shows that reversing the response options of a bipolar satisfaction question (very dissatisfied on top) leads to significantly lower reported satisfaction. In a between group experiment, 3,000 Google Opinion Rewards (Smartphone panel) respondents took a 1-question satisfaction survey. When the response options were reversed participants were 10 times more likely to select the very dissatisfied option (5% versus 0.5% prevalence). They also took 11% more time to answer the reversed scale.
The second study shows that this effect can be partially explained by respondents mistaking the word dissatisfied for satisfied. ~1750 people responded to a reversed satisfaction question in an in-product survey on fonts.google.com. In a follow-up verification question (“You selected [answer option], was this your intention?”), 42.1% of the respondents indicated that they had selected very dissatisfied by mistake. Open ended feedback suggests that respondents hadn’t read properly and expected the positive option on top.
More experiments should be conducted on different samples to better understand the interaction of scale orientation versus the type of scale (unipolar vs. bipolar).
Research Areas
