Research ArticleSCIENTIFIC COMMUNITY

NIH peer review: Criterion scores completely account for racial disparities in overall impact scores

See allHide authors and affiliations

Science Advances  03 Jun 2020:
Vol. 6, no. 23, eaaz4868
DOI: 10.1126/sciadv.aaz4868
  • Fig. 1 Multilevel NIH review structure for a hypothetical example of three applications (App. 1, 2, and 3) submitted by two PIs (yellow and red).

    Thick blue lines show structural connections. Thin lines show hypothetical assignments for the three applications. Rectangles are specified as fixed effects and ellipses as random effects in our mixed-effects models.

  • Fig. 2 Frequency histograms for the five preliminary criterion scores and the preliminary overall impact score.

    Top row: matched black (purple) and matched white (yellow) applications comparison, with overlap in orange; bottom row: all black (purple) and random white (light green) applications comparison, with overlap in dark green.

  • Table 1 Study variables.

    IPEDS, Integrated Postsecondary Education Data System, a database of survey information gathered by the Department of Education about every college or university that participates in federal financial aid programs; HBCU, historically black college or university; HSI, Hispanic-serving institution; SEP, Special Emphasis Panel. See model descriptions for variable inclusion.

    TypeNameDescription
    Dependent variablePreliminary overall impactInteger score from 1 to 9; smaller is better
    Variables of interest
      RacePI black1 for black, 0 for white; self-reported
      Preliminary criteria
    SignificanceInteger score from 1 to 9; smaller is better
    InvestigatorInteger score from 1 to 9; smaller is better
    InnovationInteger score from 1 to 9; smaller is better
    ApproachInteger score from 1 to 9; smaller is better
    EnvironmentInteger score from 1 to 9; smaller is better
    Structural covariates
      CSR peer review
    IRGIntegrated Review Group
    SRGScientific Review Group
    Institute/CenterNIH Institute/Center making funding decisions
      Other indicators
    Application IDEncrypted application indicator
    Applicant IDEncrypted applicant/PI indicator
    Reviewer IDEncrypted reviewer indicator
    Other covariates
      Applicant-specific
    GenderF/M, self-reported
    EthnicityHispanic/Latino or not, self-reported
    Career stageEarly stage (ES), experienced, or non-ES
    new investigator
    Degree typePh.D., M.D., M.D./Ph.D., Other
    Terminal degree yearYear of most recent degree
    NIH funding historyFirst NIH application, previously applied,
    or previously funded
    Geographic locationLocation of institution: central, east, south, or west
    NIH funding binFY 2014 total institution NIH funding; five bins
    Institution sectorPublic, private, or other
    Graduate education1 if institution provides graduate education; 0 if not
    IPEDS lookup1 if institution in IPEDS database; 0 if not
    MSI typeMinority-serving institution type: HBCU, HSI,
    or otherwise
      Application-specific
    Application typeNew or renewal
    Solicitation typeRequest for application, Program announcement,
    Others
    Amended statusAmended or not
    Multiple PIsYes or no
    Requested costsFunding dollars requested
    Support yearsSupport years requested, from 1 to 5
    Council year2014–2016; year of review councils
    Review group typeStanding study section, recurring SEP,
    or nonrecurring SEP
    Human subjectsAcceptable, unacceptable, or inapplicable
    Animal subjectsAcceptable, unacceptable, or inapplicable
    Child codeAcceptable, unacceptable, or inapplicable
    Gender codeAcceptable, unacceptable, or inapplicable
    Minority codeAcceptable, unacceptable, or inapplicable
  • Table 2 NIH’s descriptions for overall impact and five review criteria (48).

    ScoreDescription
    Overall impactReviewers will provide an overall impact/priority score to reflect their assessment of the likelihood for
    the project to exert a sustained, powerful influence on the research field(s) involved, in consideration
    of the following five core review criteria, and additional review criteria (as applicable for the
    project proposed).
    Scored review criteria
      SignificanceDoes the project address an important problem or a critical barrier to progress in the field? If the aims of
    the project are achieved, how will scientific knowledge, technical capability, and/or clinical practice
    be improved? How will successful completion of the aims change the concepts, methods,
    technologies, treatments, services, or preventative interventions that drive this field?
      InvestigatorsAre the PIs, collaborators, and other researchers well suited to the project? If early-stage investigators or
    new investigators are in the early stages of independent careers, do they have appropriate experience
    and training? If established, have they demonstrated an ongoing record of accomplishments that
    have advanced their field(s)?
      InnovationDoes the application challenge and seek to shift current research or clinical practice paradigms by using
    novel theoretical concepts, approaches or methodologies, instrumentation, or interventions? Are the
    concepts, approaches or methodologies, instrumentation, or interventions novel to one field of
    research or novel in a broad sense? Is a refinement, improvement, or new application of theoretical
    concepts, approaches or methodologies, instrumentation, or interventions proposed?
      ApproachAre the overall strategy, methodology, and analyses well reasoned and appropriate to accomplish the
    specific aims of the project? Are potential problems, alternative strategies, and benchmarks for
    success presented? If the project is in the early stages of development, will the strategy establish
    feasibility and will particularly risky aspects be managed?
      EnvironmentWill the scientific environment in which the work will be done contribute to the probability of success?
    Are the institutional support, equipment, and other physical resources available to the investigators
    adequate for the project proposed? Will the project benefit from unique features of the scientific
    environment, subject populations, or collaborative arrangements?
  • Table 3 Matching variables.

    NameDescription
    Applicant
      GenderF/M, self-reported
      EthnicityHispanic/Latino or not, self-
    reported
      Career stageEarly stage (ES), experienced, or
    non-ES new investigator
      Degree typePh.D., M.D., M.D./Ph.D., other
      NIH funding binFY 2014 total institution NIH
    funding; five bins
    Application
      Application typeNew or renewal
      Amended statusAmended or not
      IRGIntegrated Review Group
  • Table 4 Sampled data summary statistics by application subset.

    SubsetUnique PIsReviewersReviewsApplications
    All black5002,3102,9261,015
    Matched black4562,0842,578890
    Matched white1,4973,8664,8931,676
    Random white1,9044,4605,6692,030
    Total3,6797,90113,1404,596
  • Table 5 Selected parameter estimates from models 1 to 4.

    Race coefficient estimates, their effect sizes, and variance components estimates from four hierarchical linear models for preliminary overall impact scores fit on n = 7471 reviews of 2566 applications. Model 1 controls for structural covariates; model 2 controls for structural and applicant/application-specific covariates; model 3 controls for structural covariates and criterion scores; model 4 controls for structural and applicant/application-specific covariates and criterion scores. Control variables are listed in Table 1. Coefficient estimates for control variables are not shown. Significance * is reported for P < 0.005. In mixed-effects models, multiple effect sizes exist for a given coefficient; we report the coefficient divided by the residual SD. For more information, see (49).

    ParametersModel 1Model 2Model 3Model 4
    Race fixed effect
      Coefficient0.466*0.350*0.0100.014
      (SE)(0.062)(0.051)(0.017)(0.018)
      P<0.005<0.0050.5610.431
      Effect size0.3580.2720.0180.025
    Random effects
      Reviewer SD0.5070.5000.2860.286
      PI SD0.8830.5780.1000.082
      SRG SD0.3430.2710.0840.075
      Residual SD1.3001.2840.5650.562
  • Table 6 Selected parameter estimates, commensuration model.

    Preliminary criterion score, race, and commensuration coefficient estimates, and variance components estimates, for preliminary overall impact scores on n = 7471 reviews of 2566 applications. Control variables (coefficient estimates not shown) include structural and applicant/application-specific covariates from Table 1. Significance * is reported for P < 0.005.

    VariableEstimate (SE)P
    Fixed effects
      Significance0.258* (0.008)<0.005
      Investigator0.057* (0.011)<0.005
      Innovation0.129* (0.008)<0.005
      Approach0.598* (0.007)<0.005
      Environment0.022 (0.011)0.057
      PI race = black−0.024 (0.047)0.610
      Significance * PI black−0.034 (0.013)0.010
      Investigator * PI black0.018 (0.017)0.298
      Innovation * PI black−0.020 (0.014)0.144
      Approach * PI black0.041* (0.012)<0.005
      Environment * PI black−0.010 (0.018)0.596
    Random effects
      Reviewer intercepts SE0.286
      PI intercepts SE0.079
      SRG intercepts SE0.076
      Residual variability SE0.562

Supplementary Materials

  • Supplementary Materials

    NIH peer review: Criterion scores completely account for racial disparities in overall impact scores

    Elena A. Erosheva, Sheridan Grant, Mei-Ching Chen, Mark D. Lindner, Richard K. Nakamura, Carole J. Lee

    Download Supplement

    This PDF file includes:

    • Tables S1 to S10
    • Figs. S1 to S4
    • References

    Files in this Data Supplement:

Stay Connected to Science Advances

Navigate This Article