Chavalarias et al. 2016
To download only this data file: Chavalarias.rds (24 MB)
To download all BEAR datasets, click here.
Description
Reference: Chavalarias et al. (2016).
Research question: reporting of p-values in the biomedical literature
Data collection: the authors used large-scale text mining of MEDLINE abstracts and PubMed full texts: 4.5mln p-values in 1.6mln MEDLINE abstracts; 3.4mln p-values in 385k PubMed full-text articles.
Data availability: the extracted p-value dataset is publicly available https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/6FMTT3 (about 6GB of data) As per the hosting platform: this dataset is released under the Creative Commons Attribution-NonCommercial 4.0 International License.
Data processing: unsigned z-values were derived from p-values assuming two-sided tests. P-value truncation operators were processed by collapsing variants such as <<, <<<, <=, less than, and =< into <. We dropped 0.7% of rows where p-values did not have a “plain” format and 0.08% of rows where truncation could not be unambiguously classified. For the distributed version of BEAR 50,000 studies are selected at random, with one row per study, in order to keep the file size manageable.
Additional variables used: source indicator (abstract versus full text).
Model of z-values
| Characteristic | Estimate |
|---|---|
| Probability of significance | 52% |
| Relative probability of publication for |z| < 1.96 | 0.15 |
| Successful replication for |z| > 1.96 | 76% |
| Correct sign for |z| > 1.96 | 100% |
What do these terms mean?
- Probability of significance
- The reported value is the assurance: the proportion of significant results adjusted for publication bias.
- Relative probability of publication
- The relative probability of observing a result below the |z| = 1.96 threshold rather than above it. Values below one indicate lower observation probability below the conventional two-sided significance threshold.
- Successful replication
- The probability that an exact replication has the same sign and |z| greater than 1.96, conditional on the original result having |z| greater than 1.96.
- Correct sign
- The probability that the observed effect has the same direction as the true effect, conditional on an original result with |z| greater than 1.96.
