Research ArticleSCIENTIFIC COMMUNITY

Nonreplicable publications are cited more than replicable ones

See allHide authors and affiliations

Science Advances  21 May 2021:
Vol. 7, no. 21, eabd1705
DOI: 10.1126/sciadv.abd1705
  • Fig. 1 Distribution of total citation counts and replicability.

    The distribution of citation counts, separately depending on replicability (a P value of 0.05 or lower in a two-sided test and with an effect in the same direction as the original one), is shown separately for Nature/Science (A), Economics (B), and Psychology (C), for which replication markets were conducted, as reported in (11). CDF, cumulative distribution function.

  • Fig. 2 Total citation count and replicability measures.

    Average marginal effect on total citation count, as a function of whether the original paper was replicated (leftmost confidence interval), the relative effect size of the replication, and the market price reflecting experts’ belief that each study would replicate (between 0 and 1). Effects obtained from Poisson regressions, pooling the three replication projects for which replication markets were conducted, as reported in (11), and adding project fixed effects and robust standard errors. The regression table is provided in table S3. Bars indicate 95% confidence intervals.

  • Fig. 3 Yearly citation count by replicability.

    The average yearly citation count per year for studies that were not replicated (according to P value of the replication) in each replication study [(A) for Nature/Science, (B) for Economics, and (C) for Psychology papers in replication markets] and for those that were replicated. The light gray area shows the year(s) in which the original studies were published, and the dark line shows the year in which the replication study was published.

  • Fig. 4 Distribution of citations that papers citing a replicable versus nonreplicable publication receive.

    This figure shows the CDF of citations of citing papers, separated by whether the citing paper cited a replicable or a nonreplicable publication for each replication study [(A) for Nature/Science, (B) for Economics, and (C) for Psychology papers in replication markets]. The CDF displays 95% of the distribution (to more clearly distinguish the distributions, we exclude the upper 5% of citing papers in terms of citations as their distributions are highly skewed). The replication projects are excluded from the sample of citing papers.

  • Fig. 5 Likelihood of publication and impact factor of papers citing a replicable versus nonreplicable publication.

    This figure shows the fraction of citing papers that are published in a journal that is listed in the JCR database (A) and the average JCR impact factor (B), separated by whether the citing paper cited a replicable or a nonreplicable publication. Bars show 95% confidence intervals. The replication projects are excluded from the sample of citing papers.

  • Table 1 Yearly citation count and replicability.

    This table shows the results of a difference-in-differences Poisson regression on citations per year, as downloaded for each replicated paper through Publish or Perish in March of 2020, using a random-effects estimator. The variable Replicated is an indicator variable that takes a value of 1 if the replication study found a significant effect in the same direction as the original study. The variable After publication of replication is an indicator variable that takes a value of 1 for the years after the replication study was published. All regressions include replication project fixed effects. Column (2) also includes the following characteristics of the original study: length of the paper (number of pages), number of authors, share of male authors, whether the experiment was conducted in the United States, whether it was conducted in English, and whether it was conducted online. Standard errors are reported in parentheses. The data source for the citation counts is the software Publish or Perish in March of 2020 (23).

    (1)(2)
    Citations per year
    Replicated (P < 0.05)−15.040***−15.964***
    (5.341)(5.631)
    After publication of
    replication
    −9.065***−9.025***
    (1.167)(1.147)
    Replicated × After
    publication of
    replication
    1.2361.229
    (0.943)(0.938)
    Years since publication3.522***3.506***
    (0.361)(0.353)
    Replication project
    fixed effects
    YesYes
    Paper characteristicsNoYes
    Number of papers8080
    Observations714714

    ***P < 0.01.

    Supplementary Materials

    • Supplementary Materials

      Nonreplicable publications are cited more than replicable ones

      Marta Serra-Garcia and Uri Gneezy

      Download Supplement

      This PDF file includes:

      • Data sources and procedures
      • Additional results
      • Figs. S1 to S4
      • Tables S1 to S13
      • References

      Files in this Data Supplement:

    Stay Connected to Science Advances

    Navigate This Article