Analysis Shows No Evidence of Bias Against Fellow Nominees From Industry

IEEE subcommittee’s study shows candidates are treated ​comparably

20 November 2017

Fellow is a distinction reserved for select IEEE members whose extraordinary accomplishments in any of the IEEE fields of interest are deemed fitting. Recently, concerns have been raised about whether there is bias in the elevation-to-Fellow process, either based on a nominee’s employment type or the nomination category for which he or she is being considered. The most common concern is whether those from industry are less likely to be elevated than those from academia.

The 2016 Fellow strategic planning subcommittee performed an extensive statistical analysis on Fellow nomination and elevation data, including a study of the elevation probability of nominees as a function of society and council ranking and as a function of some nominee categories, such as employment type.

Elevation probability trend

Per IEEE bylaws, the number of elevations each year to Fellow grade cannot exceed 0.1 percent of IEEE’s voting membership.

On the one hand, the number of elevations has been somewhat stable, increasing by 20 percent between 1999 and 2017. On the other hand, nominations have grown much faster: by more than 67 percent in the same 19-year period, mostly due to an increase of academic nominees. This year 73 percent of the candidates were from academia. The number of industry nominees has remained rather constant, and this year they accounted for 19 percent. This is shown in Figure 1, below.

FACT #1: Today’s Fellow elevation probability is the lowest of the past 19 years (32 percent in 2017). Most elevations are among academics because they constitute by far the largest group.

Elevation probability by employment category

The subcommittee looked at the conditional elevation probability (CEP), where the conditioning event is the nominee’s employment type. The Fellow process tracks four employment types: industry, academia, government, and other. Table 1, below, shows the average CEP with respect to all four types is approximately the same. However, industrial nominees have had a CEP consistently higher than that of academics for nine of the past 10 years. This result contradicts the perception that the Fellow committee is somewhat biased against industry nominees.

FACT #2: The 10-year average CEP is weakly dependent on the nominee’s employment type, but industry nominees usually have a higher CEP than academic ones.

Elevation probability by nomination category

A candidate for Fellow elevation can be nominated in one of four Fellow categories: application engineer/practitioner (AE/P), educator (EDU), technical leader (TL), and research engineer/scientist (RE/S). Table 2, below, shows the CEP of nominees with respect to the four categories. Research engineer/scientist nominees have the highest elevation probability, followed closely by technical leaders. Educators and application engineers/practitioners have the lowest elevation probability—much lower than research engineers/scientists and technical leaders.

FACT #3: The 10-year average CEP is strongly dependent on the nomination category.

Are only industry scientists performing well? This question addresses the belief of many that only industry research engineers/scientists (accounting for half of all industrial nominees) perform well in Fellow elevation, while industry nominees in all other categories perform poorly, and certainly worse than academics. Table 3, below, shows this belief is also incorrect, as industry nominees perform better than academic ones in all categories. Notably, both industry research engineers/scientists and technical leaders significantly outperform their peers from academia.

FACT #4: Industry nominees perform better than academic ones in all categories.

But some questions remain:

  • Is the system unfair to application engineers/practitioners and educators?
  • Are the evaluation criteria biased?
  • If the process is not unfair or biased, how can we explain the lower elevation rates for application engineers/practitioners and educators?

Unfortunately, the data analysis cannot answer those questions. Statistical anomalies do not identify root causes. However, we believe there are many plausible interrelated causes (in addition to possible bias) that could explain why scientists (whatever their employment type) outperform non-scientists in Fellow elevation.

The set of research engineer/scientist nominees might include a larger number of qualified nominees than other categories, and that can happen simply because research engineers/scientists are more numerous than any other Fellow category (73 percent of all nominations). As a consequence, Fellow elevations (at most equal to the 0.1 percent of the IEEE voting membership) might be predominantly constituted of research engineers/scientists, leaving fewer elevations available to other Fellow categories. Increasing the number of qualified application engineer/practitioner and educator nominees could add balance.

Verifiable evidence and its impact, the two essential requirements for Fellow elevation, might be objectively more difficult to create for industrial employees. Many academics are incentivized to produce such evidence on an ongoing basis; it is part of their job. However, many practitioners might not be engaged in activities for which verifiable public evidence is available. For such achievements, there are other IEEE awards that can recognize and honor the accomplishments better than IEEE Fellow elevation.

It is arguably easier to assess the impact of the types of verifiable evidence typically produced by research engineers/scientists (such as publications with wide dissemination) than the evidence typically produced by application engineers/practitioners (such as products, patents, and trade secrets). Specifically, the impact of products and patents is more difficult to assess than that of publications, because assessment of the latter has been conducted for generations.

The incentives for becoming Fellow are often greater for academics than for members from industry. Due to competitive pressure, some companies no longer have incentives for—and might actively discourage—seeking a nomination for their employees. Therefore there are fewer industrial nominees and less verifiable evidence to support their nomination.

A 2006 study edited by IEEE Fellow Robert Lucky and Jon Eisenberg for the U.S. National Academy of Engineering, “Renewing U.S. Telecommunications Research,” supports these points. The study showed a sharp change in publication trends in United States and international telecom research between 1970 and 2004. About 80 percent of articles published in 1970 in the IEEE Transactions on Communications were written by people from industry (worldwide); by 2004 that figure had declined to 15 percent. Considering U.S. industry alone, the percentage of articles written by industry authors declined to 7 percent from 70 percent.

Granted, the chances for Fellow elevation have decreased considerably over time as many more nominees compete today for approximately the same number of elevations. However, although implicit or explicit bias against non-scientists (including educators) during the evaluation by society, council, and IEEE Fellow committees is certainly possible, there is no evidence of it.

Current initiatives

In an ongoing effort to improve the entire evaluation process for the elevation of IEEE Fellows, the IEEE Fellow committee has approved several changes that will be rolled out in the coming years.

Revisions to the process of the intermediary evaluation by societies and technical councils, including a new evaluation form, have been approved. Revisions to the nomination form are also forthcoming. Also, all evaluators now undergo a detailed orientation procedure describing the challenges of fair evaluation given the wide diversity among the nominees.

Recommendation guides have been created for all participants in the Fellow process: nominators, references, endorsers, and evaluators. The guides are on the IEEE Fellow website. We encourage all to download them and distribute them as necessary.

Reibman is 2016–2017 chair of the IEEE Fellow committee, and Galli is the committee’s 2016–2017 vice chair.

Learn More