Lesson 8 of 10 · 7 min
Bias
You know where a model’s biases come from and how to spot them in a text about people.
A model learns from huge quantities of texts written by humans. It also learns their imbalances. It can associate a profession with a gender, describe a region through clichés or know the situations most present in its data better. This is not an intention. It is a reflection, which the fluency of the text makes hard to see.
- Job ads and job descriptions.
- Evaluations, appraisals and recommendations about a person.
- The examples and characters in training materials.
- Texts about groups, countries or cultures you know little about.
Less effective
Write a job ad for a team leader position in logistics.
More effective
Write a job ad for a team leader position in logistics, open to all genders. Describe the tasks and the skills expected rather than a personality profile. Flag any wording that could discourage some of the candidates.
You describe the position rather than the person you imagine, and you ask the model to point out its own risky wording.
These biases have been measured at scale. Commercial facial recognition systems made mistakes far more often on darker-skinned women than on lighter-skinned men. Language models inherit the imbalances of their training texts in the same way.
References
- Bender, Gebru, McMillan-Major, Shmitchell (2021). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. Proceedings of ACM FAccT 2021, p. 610–623. doi.org/10.1145/3442188.3445922
- Buolamwini, Gebru (2018). Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. Conference on Fairness, Accountability and Transparency, PMLR 81, p. 77–91. proceedings.mlr.press/v81/buolamwini18a.html