Bias in Computer Systems, and Beyond

Article summary

I enjoy reading a good academic paper every now and then. Recently, I was prompted by an email from a former Atomic Object coworker, Mike Karlesky, to read through “Bias in Computer Systems” by Friedman & Nissenbaum. I recommend reading it for yourself (PDF available here).

Bias isn’t always negative, but the focus of the research presented in this paper is bias of the decidedly unfair variety:

… we use the term bias to refer to computer systems that systematically and unfairly discriminate against certain individuals or groups of individuals in favor of others. A system discriminates unfairly if it denies an opportunity or a good or if it assigns an undesirable outcome to an individual or group of individuals on grounds that are unreasonable or inappropriate.

They outline three classifications for unfair bias in computer systems: Preexisting Bias, Technical Bias, and Emergent Bias.

Types of bias

Preexisting Bias

Bias that gets built into a software system because the organization determining the requirements for the software is biased. The software reflects their own bias.

Technical Bias

Technical bias emerges as a consequence of technical constraints or decisions. Limited space on a display, input devices, and specific algorithms can bias a system in one way or another.

Emergent Bias

Use of a software system changes over time. Through the addition of new users, new types or sources of data, or a myriad of other possibilities, bias can emerge in ways that would have been difficult, if not impossible, to predict when the system was built.

Bias spotting

Since reading their work I’ve been much more aware of bias in computer systems, and pretty much everywhere else. In this case, being aware of the problem is the first step to preventing bias in the systems you build.

The following two are non-software examples I spotted in the physical world. I’m not sure physical-world biases always fit in the classification system outlined in the paper, but they’re interesting to consider.

Apple’s release of WWDC tickets for 2012

Apple opened their ticket purchasing for WWDC at a time when people on the West Coast were less likely to be awake and able to purchase tickets. They sold out very quickly, leaving many West Coast residents crying foul because of the unfair bias against them.

Getting teeth cleaned at the dentist

I noticed that the rooms used for dental cleanings are set up for right-handed people. The room can’t be easily rearranged and I imagine a left-handed dental hygienist would be at a notable disadvantage.

What examples of bias have you noticed recently, in software or elsewhere?

  • Phil Kirkham Phil Kirkham says:

    That’s 3 more biases I can add to the list of biases that testers need to be aware of – see here for more:

  • […] Bias in Computer Systems, and beyond Written by: Jason Porritt […]

  • Jason, here’s another quite accessible article in this space. We read this after I sent those original papers along.

    Can an algorithm be wrong? Twitter Trends, the specter of censorship, and our faith in the algorithms around us

    May you find it thought provoking.

  • Comments are closed.