Barnett and Wren 2019
To download only this data file: BarnettWren.rds (16 MB)
To download all BEAR datasets, click here.
Description
Reference: Barnett and Wren (2019).
Research question: Bias for statistical significance in health and medical journals
Data availability: The dataset Georgescu.Wren.RData may be downloaded from https://github.com/agbarnett/intervals/tree/master/data or https://github.com/jdwren/ASEC. The repositories give no clear licence, but the BMJ Open article is CC BY 4.0.
Data description and source: The dataset is a collection of confidence intervals for ratio estimates from Medline papers published between 1976 and 2019. Authors scraped (via regular expressions, followed by an independent check using another data mining algorithm, manual checks for 10,000 intervals) 968,000 CIs from abstracts and 350,000 from full texts.
Notes: These are “ratio estimates” such as odds ratios, hazard ratios and risk ratios. Note that binary outcomes tend to contain (much) less information than continuous outcomes.
Data processing:
We use the default approach to deriving estimates from confidence intervals (see elsewhere in the documentation). We set missing ci.level values to 0.95. Among intervals with a known confidence level, only about 0.3% are not 95%.
Of note, over 6% of rows are removed due to low symmetry of confidence intervals. These rows likely correspond to cases of non-Wald confidence intervals. However, we do not have any details of measures to investigate this further. A small fraction of a percent has other invalid rows. Rows excluded under the default CI procedure remain in the individual dataset for future investigations.
Since all measures are ratios, we use the log scale for all calculations. There are some differences between using the interval midpoint or reported point estimate as effect size. (As per default approach we do the latter.) Using the interval midpoint would change the significant proportion by under 2 percentage points in eligible pre-thinning data.
For the main version of BEAR dataset, BEAR.rds, 50,000 studies are selected at random, with one row per study, in order to keep the file size manageable.
Study characteristics:
- Study ID: we use the PubMed identifier (
pubmed). - There is no information on study design (i.e. we cannot distinguish, for example, between prospective and retrospective studies or between RCTs and observational studies).
- Effects are ratios on a logarithmic scale, without distinguishing odds, risks, hazards etc.
Model of z-values
| Characteristic | Estimate |
|---|---|
| Probability of significance | 30% |
| Relative probability of publication for |z| < 1.96 | 0.08 |
| Successful replication for |z| > 1.96 | 58% |
| Correct sign for |z| > 1.96 | 98% |
What do these terms mean?
- Probability of significance
- The reported value is the assurance: the proportion of significant results adjusted for publication bias.
- Relative probability of publication
- The relative probability of observing a result below the |z| = 1.96 threshold rather than above it. Values below one indicate lower observation probability below the conventional two-sided significance threshold.
- Successful replication
- The probability that an exact replication has the same sign and a |z| greater than 1.96, conditional on the original result having |z| greater than 1.96.
- Correct sign
- The probability that the observed effect has the same direction as the true effect, conditional on an original result with |z| greater than 1.96.
