Arel-Bundock et al 2022
To download only this data file: ArelBundock.rds (368 KB)
To download all BEAR datasets, click here.
Description
Reference: Arel-Bundock et al. (2022).
Research question: assess statistical power in political science research
Data availability: replication package (zip file with date 20241010) can be found at https://osf.io/fgdet. All data are in public domain.
Data description and source: Authors searched for, and contacted authors of, meta-analysis articles across 141 journals in political science. The resulting source dataset contains 16,649 hypothesis tests, grouped in 351 meta-analyses and reported in 46 peer-reviewed meta-analytic articles.
Notes: This is a collection of meta-analysis inputs rather than a direct extraction of all results from primary articles. The source combines a political-science search with an earlier Doucouliagos work, meaning that some of this dataset will overlap with Askarov et al. However, the two articles cover different fields and therefore BEAR keeps the two datasets separate.
Data processing: using the same code as original authors (replication package that runs on makefiles), but skipping some additional processing that they’ve done for their paper. BEAR uses question_id as metaid (351 meta-analysis/research question groups) Signed z-values were computed as reported estimates divided by their standard errors. We verified that the distributions of z-values matched with the article.
Additional variables used: BEAR release includes meta-analytic article (46 papers). The dataset also includes sample field records whether a row comes from the Doucouliagos data collection or the later Briggs/Arel-Bundock data collection.
Model of z-values
| Characteristic | Estimate |
|---|---|
| Probability of significance | 40% |
| Relative probability of publication for |z| < 1.96 | 0.64 |
| Successful replication for |z| > 1.96 | 68% |
| Correct sign for |z| > 1.96 | 99% |
What do these terms mean?
- Probability of significance
- The reported value is the assurance: the proportion of significant results adjusted for publication bias.
- Relative probability of publication
- The relative probability of observing a result below the |z| = 1.96 threshold rather than above it. Values below one indicate lower observation probability below the conventional two-sided significance threshold.
- Successful replication
- The probability that an exact replication has the same sign and |z| greater than 1.96, conditional on the original result having |z| greater than 1.96.
- Correct sign
- The probability that the observed effect has the same direction as the true effect, conditional on an original result with |z| greater than 1.96.
