Head et al 2015
Description
Reference: Head et al. (2015).
Research question: extent of p-hacking and its impact on meta-analyses.
Data collection: PubMed papers that are open access, up to 2014. Authors used regular expressions to extract p-values from text of Abstract and Results, but not tables. Downloaded data has about 220,000 studies and about 2mln rows.
Data availability: dataset available at https://datadryad.org/dataset/doi:10.5061/dryad.79d43 under CC0 1.0 Universal licence.
Data processing: we followed the same minimal clean-up steps (e.g. removing values found in large supplementary sections) as the original paper and also attached PubMed IDs to the dataset using DOI look-up.
When constructing the BEAR dataset, we derived unsigned z-values from p-values. We treated p-values recorded as exactly 0 as truncated and set the z-operator to >. We recoded all inequalities to sharp inequalities. For the distributed version of BEAR 50,000 studies are selected at random, with one row per study, in order to keep the file size manageable.
Additional variables used: we use the “Abstract vs Results” variable for grouping
Model of z-values
| Characteristic | Estimate |
|---|---|
| Probability of significance | 46% |
| Relative probability of publication for |z| < 1.96 | 0.24 |
| Successful replication for |z| > 1.96 | 74% |
| Correct sign for |z| > 1.96 | 99% |
What do these terms mean?
- Probability of significance
- The reported value is the assurance: the proportion of significant results adjusted for publication bias.
- Relative probability of publication
- The relative probability of observing a result below the |z| = 1.96 threshold rather than above it. Values below one indicate lower observation probability below the conventional two-sided significance threshold.
- Successful replication
- The probability that an exact replication has the same sign and |z| greater than 1.96, conditional on the original result having |z| greater than 1.96.
- Correct sign
- The probability that the observed effect has the same direction as the true effect, conditional on an original result with |z| greater than 1.96.
