IEEE Standard to Rate the Trustworthiness of News Sites

Automated rating system to analyze factual accuracy, headlines, and more

27 March 2018

Reputable news outlets, whether they be traditional newspapers or social media platforms such as Facebook, are looking for ways to stem the spread of false and misleading articles. The IEEE Standards Association is trying to help readers detect such content with its new IEEE P7011 Standard for the Process of Identifying and Rating the Trustworthiness of News Sources.

The standard will be an open, automated system of easy-to-understand ratings, according to IEEE Student Member Joshua Hyman, chair of the IEEE P7011 working group. He says the standard will score a representative sample of articles by analyzing a variety of factors including the headline, the organization’s use of retractions, bias, factual accuracy, and native advertising—paid content written to look like a journalistic piece.

The news outlet will receive a rating in the form of a letter grade, telling readers just how trustworthy the source is, he says. The goal is to inform the consumer as to the quality of the news purveyor, not fact check an article like Snopes, for example.

Hyman is in charge of vendor relations at the University of Pittsburgh, where he is pursuing a master’s degree in public policy. The IEEE P7011 project is based on his research article, “Addressing Fake News: Open Standards and Easy Identification,” which is available in the IEEE Xplore Digital Library.

The Institute asked Hyman about how the rating system would operate, concerns about censorship, and preventing bias.

How will IEEE P7011 work?

At a very high level, the standard will look at a news purveyor’s content, presentation, and policies. It will check five aspects: the factual accuracy of the information being presented, misleading headlines (better known as clickbait), utilization of retraction policies, whether there is bias and whether there is a clear distinction between advertisements and news articles.

It will not analyze every article—that would be too big a task. Instead, it would analyze a random selection to provide a standard baseline.

Detecting language bias is easy to automate. For example, the system would look at the number of times an article includes phrases like claimed, versus stated. There are also text analytics that detect bias. The working group will develop data sets for this.

The standard will also examine link networking—essentially what’s linking to what. This can help us develop a measurement for accuracy.

Some degree of fact-checking could also be automated. Fact and truth are not necessarily the same thing. You can evaluate a fact as true or false, but the presentation of the fact can change whether it’s truthfully presented. Something can be taken out of context or omitted, and that can affect how information is presented.

With regard to misleading headlines, the system can conduct a textual analysis of what a headline says, versus what kind of word sets are used in the article. There are sensationalized headlines that don’t have anything do with the article’s content. That’s when you have to ask: Is this purveyor’s website just trying to get you to click on a headline for the sake of clicks, or because the headline is pertinent or informative?

To some extent, the analysis of advertisements also can be automated. We’ve seen a lot more native advertisements than ever before. These often are pro-industry articles written by a company that has a special interest, but it’s not always clear to the reader that it’s a paid advertisement masking as a journalistic piece. The system can look at how the pages are coded or look for keywords that inform the reader they’re looking at an advertisement.

The rating system will give the news site a letter grade, using the traditional A through F rating most people are familiar with from school. This will help the reader see that an article is from a site that has a good, bad, or average reputation. Ideally, what we hope to do with this standard is to slow down or stop the viral spread of misinformation. The random sample-set of articles will inform the grade, and new articles will be sampled on a regular basis to create a moving average.

What industries is this rating system targeting?

Social media platforms, like Facebook, and search engines, like Google. But it’s also going to include news outlets, like The New York Times and Fox News. These companies are welcome to be members of the working group so they could help build some of the language sets, which will continue to evolve. It is our hope that media platforms will adopt the standard.  

Why is IEEE interested in developing such a standard?

An IT solution is what’s needed to solve this problem, and it can’t be done by a government or private company. In the United States, that would be a First Amendment issue, and impractical in the current political climate. It also can’t be solved by private companies, because there is always going to be some aspect of distrust.

That leads us to a nonprofit, non-political institution that is in the technology space. IEEE is the largest and most respected one of these organizations. What we are trying to do is restore a semblance of trust in news-sharing sites and the institution of journalism.

What barriers prevented the spread of misinformation in the past?

There has been “fake news” since before the Internet was created, but we are seeing a completely new level with social media in the past 10 years and the development of the Internet over the past 30 years.

The barriers that existed before involved physical capital. A newspaper was not cheap to operate. The very nature of such a large investment meant that the news enterprise could not afford to appeal to only fringe ideologies. Now someone with a laptop using WordPress software can publish articles for what would have cost a fortune in the past.

In the United States, we have claims of “fake news” being issued by the highest office in the country. There are political arguments being made based on inaccurate information. News is being manipulated by foreign agents. It’s come to the point where every search engine and social media giant is scrambling to figure out a way to deal with it. The problem has become different, larger, and much more significant.

Isn’t it possible that this standard could lead to censorship by the working group members?

This standard isn’t designed to block or censor anyone. The goal is to evaluate and present an analysis of a news purveyor’s site so people have some idea of who’s telling them what.

Currently readers are expected to research the article themselves to verify whether the information is factual, or make sure there are multiple sources cited. But it’s an unrealistic expectation for the average person, who may not have the time, motivation, or skill set to do that research.

For example, we already have websites, like Angie’s List, to rate contractors we want to hire. We should have something similar that tells us whether the news organization is reputable.

How would the working group prevent bias from creeping into the code?

It’s a completely open standard—anyone can join the working group. The idea is that everyone will be able to see how these decisions are being made and provide input. We want to get a diverse set of opinions to avoid political bias. Also, the standard will continue to evolve, so any bias that might creep in would eventually be removed.

To participate, visit the IEEE P7011 Working Group Web page. The inaugural meeting will be held 13 April from 3 to 4:30 p.m. EDT via WebEx.

IEEE membership offers a wide range of benefits and opportunities for those who share a common interest in technology. If you are not already a member, consider joining IEEE and becoming part of a worldwide network of more than 400,000 students and professionals.

Learn More