Arel-Bundock et al 2022
To download only this data file: ArelBundock.rds (368 KB)
To download all BEAR datasets, click here.
Description
Reference: Arel-Bundock et al. (2022).
Research question: assess statistical power in political science research
Data availability: The replication package (a zip file dated 20241010) is available at https://osf.io/fgdet. All data are in the public domain.
Data description and source: The authors searched for meta-analytic articles across 141 journals in political science and contacted their authors. The resulting source dataset contains 16,649 hypothesis tests, grouped in 351 meta-analyses and reported in 46 peer-reviewed meta-analytic articles.
Notes: This dataset collects meta-analysis inputs, rather than repeating data extraction from individual studies. It combines an earlier dataset from Doucouliagos et al with another search. Therefore, some of this dataset will overlap with Askarov et al. However, the two articles aim to cover different fields and BEAR keeps the two datasets separate.
Data processing: We used the same code as the original authors (a replication package that uses makefiles), but skipped some additional processing they performed for their paper. BEAR uses question_id as metaid (351 groups defined by meta-analysis or research question). Signed z-values were computed as reported estimates divided by their standard errors. We verified that the distributions of z-values matched those in the article.
Study characteristics:
- Study ID: we use author’s study ID; for meta-analysis ID we use
question_id. subsetcolumn is the sourcemeta_id- We do not have information on the study design or measure of effect size.
Model of z-values
| Characteristic | Estimate |
|---|---|
| Probability of significance | 40% |
| Relative probability of publication for |z| < 1.96 | 0.64 |
| Successful replication for |z| > 1.96 | 68% |
| Correct sign for |z| > 1.96 | 99% |
What do these terms mean?
- Probability of significance
- The reported value is the assurance: the proportion of significant results adjusted for publication bias.
- Relative probability of publication
- The relative probability of observing a result below the |z| = 1.96 threshold rather than above it. Values below one indicate lower observation probability below the conventional two-sided significance threshold.
- Successful replication
- The probability that an exact replication has the same sign and a |z| greater than 1.96, conditional on the original result having |z| greater than 1.96.
- Correct sign
- The probability that the observed effect has the same direction as the true effect, conditional on an original result with |z| greater than 1.96.
