Barnett and Wren 2019
To download only this data file: BarnettWren.rds (16 MB)
To download all BEAR datasets, click here.
Description
Reference: Barnett and Wren (2019).
Research question: Bias for statistical significance in health and medical journals
Data availability: The dataset Georgescu.Wren.RData may be downloaded from https://github.com/agbarnett/intervals/tree/master/data and also https://github.com/jdwren/ASEC. Data file is Georgescu.Wren.RData. Availability: no clear licence given in the repo but the article is CC BY 4.0 in BMJ Open.
Data description and source: The dataset is a collection of confidence intervals for ratio estimates from Medline papers from 1976-2019. Authors scraped (via regular expressions, followed by an independent check using another data mining algorithm, manual checks for 10,000 intervals) 968,000 CIs from abstracts and 350,000 from full texts.
Notes: These are “ratio estimates” like odds ratios, hazard ratios and risk ratios. Note that binary outcomes tend to have (much) lower information than continuous outcomes.
Data processing: We excluded a small fraction of data. First we restricted to 95% confidence intervals only: where ci.level was missing we assumed it was 0.95; among intervals with a known ci.level, about 0.3% were not 95% and were dropped. We dropped intervals with non-positive width (i.e. lower >= upper). We used log scale (we replaced zero or negative lower bounds with a small positive constant; this affected about 0.1% of the sample) and backed out an approximate standard error and point estimate under the usual normal approximation for a 95% CI, setting point estimate to half-point of the interval.
For the distributed version of BEAR 50,000 studies are selected at random, with one row per study, in order to keep the file size manageable.
Model of z-values
| Characteristic | Estimate |
|---|---|
| Probability of significance | 30% |
| Relative probability of publication for |z| < 1.96 | 0.08 |
| Successful replication for |z| > 1.96 | 58% |
| Correct sign for |z| > 1.96 | 98% |
What do these terms mean?
- Probability of significance
- The reported value is the assurance: the proportion of significant results adjusted for publication bias.
- Relative probability of publication
- The relative probability of observing a result below the |z| = 1.96 threshold rather than above it. Values below one indicate lower observation probability below the conventional two-sided significance threshold.
- Successful replication
- The probability that an exact replication has the same sign and |z| greater than 1.96, conditional on the original result having |z| greater than 1.96.
- Correct sign
- The probability that the observed effect has the same direction as the true effect, conditional on an original result with |z| greater than 1.96.
