Is Peer Review Stifling Innovation at NIH?

With the visionary language of large federal initiatives like the “Cancer Moonshot” or provocative branding such as “NIH…Turning Discovery into Health®” and the National Institute of Health website further touting “revolutionary ideas often come from unexpected directions,” one might assume an equally ambitious approach is being taken to ensure federal life sciences research is going toward research with the most promise for positive impact and scientific advancement.  One of the calls of Vice President Joe Biden’s recent Cancer Moonshot report was urging not to accept a “business as usual” approach. A recent report underscores why that may be a problem at the National Institutes of Health (NIH).

What is the most significant factor associated with whether or not a R01 research proposal receives funding after going through NIH’s vaulted peer review system?  Would you guess the significance of the research? The innovation incorporated in the study? The experience and educational credentials of the researchers? Perhaps, the environment and or facilities in which the research will be undertaken? The process or approach used to conduct the research?

If you selected the last option, the process or approach – the ability of the researchers to conform to the established, logical, linear incrementalism expected – then you were right.  In a study of 123,000 research awards from the NIH, among all of the criteria used through the peer review scoring system, how well a R01 proposal scored in the approach criterion was found to be most strongly predictive of a funding decision, according to the June 2016 paper, How Criterion Scores Predict the Overall Impact Score and Funding Outcomes for National Institutes of Health Peer-Reviewed Applications. The authors, MK Eblen, RM Wagner, D Roy Chowdhury, KC Patel, and K Pearson are from the NIH Office of Extramural Research (OER).

The criteria of “significance” and “innovation” tied for a distant second in association with successful funding outcomes. 

Michael Lauer, NIH’s director of extramural research, advises biomedical researchers who might submit proposals for funding in the future:

“We think it’s helpful for R01 applicants to know that the description of the experimental approach is the most important predictor of funding, followed by the significance of the study. As an applicant, familiarizing yourself with the peer reviewer guidance and questions they are asked about approach and significance may be helpful as you put together your application.”

The review guidance for approach includes the following for R01 awards:

 Are the overall strategy, methodology, and analyses well-reasoned and appropriate to accomplish the specific aims of the project?

Have the investigators presented strategies to ensure a robust and unbiased approach, as appropriate for the work proposed?

Are potential problems, alternative strategies, and benchmarks for success presented?

If the project is in the early stages of development, will the strategy establish feasibility and will particularly risky aspects be managed?

Have the investigators presented adequate plans to address relevant biological variables, such as sex, for studies in vertebrate animals or human subjects?

Measures of reasonableness, appropriateness, and riskiness seem likely to favor convention, familiarity and incrementalism. So, perhaps, significance and innovation would be lower by default of the overall guidance leaning toward the expected. For instance, the significance criterion begins with the more ambitious “Does the project address an important problem or a critical barrier to progress in the field?”

In all fairness to NIH, it should be noted that the OER mission statement is focused more on process than on outcomes. “OER provides the corporate framework for NIH research administration, ensuring scientific integrity, public accountability, and effective stewardship of the NIH extramural research portfolio.” [Emphasis added.] Outcomes and impact are noticeably absent.

The report, and Biden’s report, raise questions about whether any difference will occur when the most significant factor in predicting funding is whether the description of the approach is clear and when the mission of the organization focuses on process rather than outcomes.