Jager and Leek 2014
To download only this data file: JagerLeek.rds (331 KB)
To download all BEAR datasets, click here.
Description
Reference: Jager and Leek (2014).
Research question: estimating science-wise false discovery rate.
Data collection: authors used a custom program to extract p-values from scraped PubMed abstracts for papers published in 5 main medical journals 2000-2010. 15,653 p-values are available in 5,322 articles.
Data availability: the data file pvalueData.rda is available https://github.com/jtleek/swfdr License for the programs in that repository is GNU GPL, although a license for the dataset is not stated, as far as we are aware.
Data processing: there was only minimal processing. We derived unsigned z values from p-values assuming they were two-sided.
Notably, large proportion of p-values is truncated, almost always at 0.0001, 0.001, 0.01, or 0.05. As in all datasets, we retained information on truncation. We also treated p-values recorded as exactly 0 as truncated (z-operator >). We created a crude flag for RCTs by searching the paper titles for “randomized”, “randomised” and “controlled”.
Model of z-values
| Characteristic | Estimate |
|---|---|
| Probability of significance | 45% |
| Relative probability of publication for |z| < 1.96 | 0.18 |
| Successful replication for |z| > 1.96 | 72% |
| Correct sign for |z| > 1.96 | 99% |
What do these terms mean?
- Probability of significance
- The reported value is the assurance: the proportion of significant results adjusted for publication bias.
- Relative probability of publication
- The relative probability of observing a result below the |z| = 1.96 threshold rather than above it. Values below one indicate lower observation probability below the conventional two-sided significance threshold.
- Successful replication
- The probability that an exact replication has the same sign and |z| greater than 1.96, conditional on the original result having |z| greater than 1.96.
- Correct sign
- The probability that the observed effect has the same direction as the true effect, conditional on an original result with |z| greater than 1.96.
