Can Computers Analyze Emotion Better Than People?

A computer system developed by MIT researchers can tell a lot about your smile

9 July 2012

Computers have us beat in many ways. They perform complex math equations much faster, can apparently find Waldo more quickly, and—who knew—they can even beat us in Jeopardy! as Watson proved last year. 

But one area I would never have expected a computer to beat a human in is the ability to judge our emotions. Yes, they can calculate the world’s most complex equations, but surely computers can’t judge a person’s mood better than I can after looking at them. Or so I thought.

As it turns out, computers win in this situation as well. A group of MIT researchers have developed a computer system that can do a better job of judging whether a person’s smile is out of joy or frustration. Had no clue you smiled when you were frustrated? Me neither, but according to the results of a study the researchers used to develop the system, we do. A paper on their work was recently published in the IEEE Transactions on Affective Computing.

The researchers—who include IEEE Fellow Rosalind Picard, a professor of media arts and sciences—conducted experiments in which participants were first asked to act out expressions of happiness and frustration as webcams recorded them. They were next asked to either fill out an online form designed to aggravate them (which deliberately deleted the information they had entered in a lengthy form after they hit the submit button), or to watch a video aimed at eliciting a pleasant response, such as one of a cute baby. Again, their responses were videotaped. The study participants were later shown images of each others’ smiles and asked to distinguish which were out of joy or frustration.

When asked to pretend they were frustrated, 90 percent didn’t smile. But when they were actually frustrated—by the buggy survey—the same percentage smiled. [See the photo above. The smile on the right is out of frustration]. The researchers found that happy smiles often built up gradually, but frustrated smiles appeared and faded away quickly. They then programmed a computer to differentiate between these two types of smiles. While the computer had no problem with this, only 50 percent of human participants were able to differentiate between a smile of happiness and one of frustration. Check out this video that explains their work.

So what is the point of this research, besides to bum us out that computers are better at yet another task? There are numerous possible applications. The work could lead to computers that can assess the emotional states of their users and somehow respond to their moods. And the system they developed also could be used to provide training for people who have difficulty recognizing expressions.

What do you think about the applications of this research? In which areas do you think humans will always beat computers?


Photos: Hogue et al.

IEEE membership offers a wide range of benefits and opportunities for those who share a common interest in technology. If you are not already a member, consider joining IEEE and becoming part of a worldwide network of more than 400,000 students and professionals.

Learn More