Head et al 2015
Description
Reference: Head et al. (2015).
Research question: extent of p-hacking and its impact on meta-analyses.
Data collection: The source comprises open-access PubMed papers published up to 2014. The authors used regular expressions to extract p-values from the text of the Abstract and Results sections, but not from tables. The downloaded data contain about 220,000 studies and about 2 million rows.
Data availability: The dataset is available at https://datadryad.org/dataset/doi:10.5061/dryad.79d43 under a CC0 1.0 Universal licence.
Data processing: We followed the same minimal clean-up steps (e.g. removing values found in large supplementary sections) as the original paper and also attached PubMed IDs to the dataset using DOI lookup. The article DOIs were supplied in the source data; we obtained the PubMed IDs through Europe PMC. Both identifiers are retained, with doi duplicating the source’s first.doi field.
When constructing the BEAR dataset, we derived unsigned z-values from p-values. We treated p-values recorded as exactly 0 as truncated and set the z-operator to >. We recoded all inequalities to sharp inequalities. For the distributed version of BEAR 50,000 studies are selected at random, with one row per study, in order to keep the file size manageable.
Study characteristics:
Study ID: we use the PubMed identifier (
pmid).Topic: we use the journal-level Field-of-Research classification (
Category).subsetdistinguishes abstract extraction from full text/results extraction.The dataset does not have study designs.
The extracted p-values are converted to z-values.
We do not know the underlying effect measures.
subsetdistinguishes abstract extraction from results-text extraction.
Model of z-values
| Characteristic | Estimate |
|---|---|
| Probability of significance | 46% |
| Relative probability of publication for |z| < 1.96 | 0.24 |
| Successful replication for |z| > 1.96 | 74% |
| Correct sign for |z| > 1.96 | 99% |
What do these terms mean?
- Probability of significance
- The reported value is the assurance: the proportion of significant results adjusted for publication bias.
- Relative probability of publication
- The relative probability of observing a result below the |z| = 1.96 threshold rather than above it. Values below one indicate lower observation probability below the conventional two-sided significance threshold.
- Successful replication
- The probability that an exact replication has the same sign and a |z| greater than 1.96, conditional on the original result having |z| greater than 1.96.
- Correct sign
- The probability that the observed effect has the same direction as the true effect, conditional on an original result with |z| greater than 1.96.
