Nonreplicable publications are cited more than replicable ones
- Date Posted:
- Is Database:
- Database
Research in @ScienceAdvances finds papers that fail to replicate receive more citations than those that do replicate, even after publication of replication failures.
New paper from Science Advances highlights the replication problem in science, “….We use publicly available data to show that published papers in top psychology, economics, and general interest journals that fail to replicate are cited more than those that replicate. This difference in citation does not change after the publication of the failure to replicate. Only 12% of post replication citations of nonreplicable findings acknowledge the replication failure. Existing evidence also shows that experts predict well which papers will be replicated. Given this prediction, why are nonreplicable papers accepted for publication in the first place? A possible answer is that the review team faces a trade-off. When the results are more “interesting,” they apply lower standards regarding their reproducibility….”
Core result, “…Our main finding is that papers that fail to replicate in (5-7) are cited more than those that are replicable. We find no significant change in citation trends, even after the publication of the failed replication. Notably, only a minority of publications after the failed replications were published acknowledge the failure. The “quality” of citations of papers that failed to replicate is similar to that of papers that were replicated: We do not find a difference in how often the citations of nonreplicable publications are cited by others or in the impact factor of the journals in which citations are published…”
The evidence, "...Figure 1 shows the distribution of total citation counts by the end of 2019 of the papers included in the replication projects, separately depending on replicability for Nature/Science, economics journals, and psychology journals. Replicability is measured according to the criterion that the replication project featured a P value of 0.05 or lower in a two-sided test, with an effect in the same direction as the original test….”

Marta Serra-Garcia and Uri Gneezy, "Nonreplicable publications are cited more than replicable ones,"Science Advances, May 21, 2021, https://advances.sciencemag.org/content/7/21/eabd1705




