Verification categories used to assess claims.

Proportion of claims in broad reproducibility assessment categories before and after experimental validation by the ReproSci project.

See description of each category in Material and Methods. In this article, we are using indirect reproducibility, also called conceptual replicability, to assess the validity of a claim using alternate methods to those used in the original study.

Flow of the study.

Claims were first classified in five categories based on literature. 45 unchallenged claims were selected to be experimentally tested and classified either as challenged or as verified. We further classified some unchallenged as consistent or inconsistent depending on their consistency with current knowledge.

Percentage of claims by category according to journal-impact and institution ranking.

(A, B) The distribution of claims across the five assessed categories for articles published in low-impact, high-impact, and trophy journals. The number of claims is indicated at the top. Panel A presents raw counts, while Panel B shows normalized data. (C) The distribution of claims across the five assessed categories according to the ranking of the last author institution in Shanghai Ranking’s 2010 Academic Ranking of World Universities. The number of claims is indicated at the top, while the number of institutions in each category is indicated at the bottom. The Not Ranked categories encompass research institutes (ex. CNRS), hospitals and other institution that are not included in the Shanghai ranking. In this figure, and in the rest of the paper, the number nc indicates the number of claims in each category, while the number at the bottom highlight the count in each category (e.g there are 11 high-impact journals, and 73 Top-50 institutions).

Evolution of claim replicability over time.

(A, B) The distribution of claims by categories of assessment over time is shown in A, and the proportion normalized over the total number of claims per time period is shown in B.

Distribution of irreplicability according to first authors.

A) Distribution of irreplicable claims among first author. We note that all irreplicable claims are produced by less than 20% of the first authors. Both panel includes 289 first authors contributing 991 claims from 394 articles. Six articles containing 15 claims were excluded because the first author was not identified. B) Distribution of irreplicability for all first authors. Each dot represents an author, ranked according to the proportion of challenged claims. The size of the circle indicates the number of claims and the color the percentage of verified claims. Authors tied on challenged claim proportion are ordered by verified claim proportion.

Influence of parameters associated with first author on replicability.

A) Repartition of claims in each assessment category according to the gender of the first authors. The gender of four authors, contributing 10 claims, could not be inferred based on first name ambiguity explaining why the total number of authors is 285. B) Repartition of claims in each assessment category according to status of the first authors. The stages of career of 279 first authors were obtained by interviewing last authors, querying Linkedln public profiles or extrapolated according to the publication list. Senior staff refers to permanent position scientist working in a lab. ND stands for Not Determined. C) Proportion of challenged claims as a function of the number of articles published by each of the 289 identified first authors. Coincident observations are combined into a circle. Circle area and the number shown indicate the number of first authors represented. Color indicates the percentage of claims classified as verified.

Distribution of irreplicability according to last authors (PI).

A) Distribution of irreplicable claims among leading authors. Both panels include all 156 last authors, representing 1,006 claims from 400 articles. B) Distribution of irreplicability for all leading authors. Each dot represents a leading author, ranked according to the proportion of challenged claims. The size of the circle indicates the number of claims and the color the percentage of verified claims. Authors tied on challenged claim proportion are ordered by verified claim proportion.

Influence of various parameters associated with the last author on replicability.

A) Distribution of claims by category according to the gender of all 156 last authors. B) Distribution of claims by category according to the seniority of the last authors. Senior PI were defined as having published a last author article at least 5 years before the considered articles. The total of junior and senior authors is higher than the total number of last authors as 23 PIs published both as junior and senior. 11 claims from four articles are excluded because they lacked a junior/senior classification. C) Proportion of challenged claims as a function of article count for all 156 last authors. Authors with identical coordinates are combined into a single circle; circle size and internal labels indicate the number of leading authors represented.

Patterns of irreplicability by last author according to year of entry in the field and research style.

A. The proportion of challenged claims by last author is shown according to the time at which the author published their first paper as a first or last authors in the field. It includes all 156 PIs and 1,006 claims; 13 PIs entered the field before 1995 and contributed 227 claims, including 6 challenged claims; 143 entered in 1995 or later and contributed 779 claims, including 63 challenged claims. The increase in irreplicability is associated with PIs whose first publication in the Drosophila immunity field, as either a first or last author, occurred in 1995 or later. Circle size indicates the number of published articles by each author. B. Distribution of claims by last author according to whether they have published at least one first author article on Drosophila immunity in another laboratory. Prior-experience status was available for 146 PIs. Ten PIs whose first-author publications occurred after they had already become PIs were excluded from comparison. C. Distribution of claims according to research style for all 156 PIs. Proportion of challenged claims by PIs who have continuously work on the Drosophila immunity (‘continuity PIs’) or made a transient incursion into the field (‘exploratory PIs’).

Multivariable analysis of predictors of claim irreplicability.

The forest plot reports the odds ratio of each covariate effect on a claim becoming challenged. The multivariate analysis included 869 claims from 345 articles, including 60 challenged claims, contributed by 251 first authors and 140 last authors. Points report adjusted odds ratios and horizontal lines show 94% highest-density intervals.