Jager and Leek 2014
To download only this data file: JagerLeek.rds (331 KB)
To download all BEAR datasets, click here.
Description
Reference: Jager and Leek (2014).
Research question: estimating science-wise false discovery rate.
Data collection: The authors used a custom program to extract p-values from scraped PubMed abstracts for papers published in five leading medical journals between 2000 and 2010. The dataset contains 15,653 p-values from 5,322 articles.
Data availability: The data file pvalueData.rda is available from https://github.com/jtleek/swfdr. The programs in that repository are licensed under the GNU GPL, although, as far as we are aware, no licence is stated for the dataset.
Data processing: There was only minimal processing. We derived unsigned z-values from p-values assuming they were two-sided.
Notably, a large proportion of p-values are truncated, almost always at 0.0001, 0.001, 0.01, or 0.05. We keep the truncation information and treat p-values recorded as exactly 0 as truncated (z-operator >). We created a crude flag for RCTs by searching the paper titles for “randomized”, “randomised” and “controlled”.
Study characteristics:
- Study ID: we use the PubMed identifier (
pubmedID). Additionally, article DOIs were obtained from Europe PMC. - Study design is not avaialble, but 30% of rows are flagged as RCTs based on scanning publication titles.
- The extracted p-values are converted to z-values.
- We do not know the underlying effect measures.
Model of z-values
| Characteristic | Estimate |
|---|---|
| Probability of significance | 45% |
| Relative probability of publication for |z| < 1.96 | 0.18 |
| Successful replication for |z| > 1.96 | 72% |
| Correct sign for |z| > 1.96 | 99% |
What do these terms mean?
- Probability of significance
- The reported value is the assurance: the proportion of significant results adjusted for publication bias.
- Relative probability of publication
- The relative probability of observing a result below the |z| = 1.96 threshold rather than above it. Values below one indicate lower observation probability below the conventional two-sided significance threshold.
- Successful replication
- The probability that an exact replication has the same sign and a |z| greater than 1.96, conditional on the original result having |z| greater than 1.96.
- Correct sign
- The probability that the observed effect has the same direction as the true effect, conditional on an original result with |z| greater than 1.96.
