Bias in Code is a Problem That Cannot Be Ignored

When Google Photos mistook a person for an animal, it was apparent the programmers had not considered diversity

17 May 2016

The Google Photos service stores and automatically organizes images into groups of, say, a particular individual or of dogs or cats. So when the service labeled a user’s African-American friend as an animal “so cliché he doesn’t even want to say which creature it was,” according to an article on NPR, he was horrified.

Some people blame the software for the mistake, but the question then becomes: Who developed the software? Google claims it tested the service on employees of different races and ethnicities but acknowledges the image-labeling technology is a work in progress, according to NPR. The company’s 2015 diversity report shows there are few African-Americans among its workers, though, and that might be the root of the problem.

“If the sample base of testers is of those who live in Silicon Valley, compared with a location that reflects much greater diversity, like New York City, the software will not learn the full spectrum of what a human looks like,” says Jim Isaak, IEEE life senior member and 2015 vice president of the IEEE Society on Social Implications of Technology. “In this respect, it would be useful to have an archive of photos of millions of people that could be used to train the software.”

There absolutely is bias in code and for a number of reasons, Isaak says. One is simply ignorance.


A basic example of bias, he says, is the subscription field found on many websites and apps that require users to type in their name. People from an Asian culture, for example, list their surname first rather than last. People from some cultures include multiple middle names. Others add a middle initial. Often the subscription box cannot accommodate those names.

“Allowing for these variations is something that requires knowledge and awareness of diverse cultures,” Isaak says. “That’s how biases get built into these features.”

A program such as Google Photos, however, presents a more complicated problem, because the service learns on its own based on the data it is fed. The solution is to train such programs not to make assumptions, but rather smarter decisions.

Biases also can be found when searching the Web. When browsing for job opportunities on a search engine, lower-paying jobs tend to be displayed to women. That’s not because the programmers designed the search engines that way; it’s because as more women click on lower-paying positions such as housekeepers and home health aides, the machine starts to determine that such listings are relevant to all women. It uses a variety of data it collects on users, including gender, to make such decisions. The machine forms biases on its own based on people’s collective behaviors online, Isaak says.

To correct that problem, either the programmer or the software needs to be better trained depending on the issue. Isaak suggests that companies set guidelines on how to reduce bias when writing code. Quality control reviews could be conducted to help ensure the software meets the criteria, he adds.

Learn More