Fairness Preferences, Actual and Hypothetical: A Study of Crowdworker Incentives

How should we decide which fairness criteria or definitions to adopt in machine learning systems? To answer this question, we must study the fair- ness preferences of actual users of machine learn- ing systems. Stringent parity constraints on treat- ment or impact can come with trade-offs, and may not even be preferred by the social groups in question (Zafar et al., 2017). Thus it might be beneficial to elicit what the group’s prefer- ences are, rather than rely on a priori defined mathematical fairness constraints. Simply asking for self-reported rankings of users is challenging because research has shown that there are often gaps between people’s stated and actual prefer- ences(Bernheim et al., 2013). This paper outlines a research program and ex- perimental designs for investigating these ques- tions. Participants in the experiments are invited to perform a set of tasks in exchange for a base payment—they are told upfront that they may receive a bonus later on, and the bonus could de- pend on some combination of output quantity and quality. The same group of workers then votes on a bonus payment structure, to elicit preferences. The voting is hypothetical (not tied to an outcome) for half the group and actual (tied to the actual payment outcome) for the other half, so that we can understand the relation between a group’s actual preferences and hypothetical (stated) pref- erences. Connections and lessons from fairness in machine learning are explored.

• Machine Intelligence

• Responsible AI