Arel-Bundock et al 2022
To download only this data file: ArelBundock.rds (368 KB)
To download all BEAR datasets, click here.
Description
Reference: Arel-Bundock et al. (2022).
Research question: assess statistical power in political science research
Data availability: The replication package (a zip file dated 20241010) is available at https://osf.io/fgdet. All data are in the public domain.
Data description and source: The authors searched for meta-analytic articles across 141 journals in political science and contacted their authors. The resulting source dataset contains 16,649 hypothesis tests, grouped in 351 meta-analyses and reported in 46 peer-reviewed meta-analytic articles.
Notes: This dataset collects meta-analysis inputs, rather than repeating data extraction from individual studies. It combines an earlier dataset from Doucouliagos et al with another search. Therefore, some of this dataset will overlap with Askarov et al. However, the two articles aim to cover different fields and BEAR keeps the two datasets separate.
Data processing: We used the same code as the original authors (a replication package that uses makefiles), but skipped some additional processing they performed for their paper. BEAR uses question_id as metaid (351 groups defined by meta-analysis or research question). Signed z-values are calculated as reported estimates divided by their standard errors. We verified that their distribution matched the published histogram in Arel-Bundock et al. (2022)
Study characteristics:
- Study ID: we use author’s study ID; for meta-analysis ID we use
question_id. subsetis the source article’smeta_id; each article may contain several groupings into meta-analysestopicis classification from the original dataset into American Politics, Comparative Politics, International Relations, Public Administration and Political Economy (subfieldin original data)topic_detailedis our more detailed subject classification and is available only indata/ArelBundock.rds.doi_collectionidentifies the source publication containing a collection of meta-analyses, not an individual meta-analysis or study.doi_studyis the individual paper DOI, reconstructed by matching labels within the dataset to bibliographies in 46 collections and cross-referencing them. However, it is only available for a minority of studies.ssis the number of observations; it uses thesample_sizevariable in the original dataset, rounded to an integer where positive; three zeros are treated as missing.- We do not have information on the study design or measure of effect size.
There are at least several other useful columns in the original replication package, which are included in ArelBundock.rds without any modifications. In particular, overall_effect and subset, iv and dv carry a lot of information about what is being estimated.
Model of z-values
| Characteristic | Estimate |
|---|---|
| Probability of significance | 40% |
| Relative probability of publication for |z| < 1.96 | 0.64 |
| Successful replication for |z| > 1.96 | 68% |
| Correct sign for |z| > 1.96 | 99% |
What do these terms mean?
- Probability of significance
- The reported value is the assurance: the proportion of significant results adjusted for publication bias.
- Relative probability of publication
- The relative probability of observing a result below the |z| = 1.96 threshold rather than above it. Values below one indicate lower observation probability below the conventional two-sided significance threshold.
- Successful replication
- The probability that an exact replication has the same sign and a |z| greater than 1.96, conditional on the original result having |z| greater than 1.96.
- Correct sign
- The probability that the observed effect has the same direction as the true effect, conditional on an original result with |z| greater than 1.96.
