Brodeur et al. 2024
To download only this data file: Brodeur.rds (819 KB)
To download all BEAR datasets, click here.
Description
Reference: Brodeur et al. (2024).
Research question: are preregistration and pre-analysis plans associated with reduced p-hacking and publication bias in economics?
Data collection: manually extracted test statistics from articles identified as RCT studies in leading economics journals published 2018-2021. Extraction was performed table by table. They collected coefficients of interest from results tables, excluding constants, balance checks, robustness checks, regression controls, and placebo tests. Each source row corresponds to a single reported test statistic, linked to a paper-level identifier. Unlike most other sources in BEAR, this means the source data have dozens of estimates per paper. The paper reports 314 articles and 15,992 test statistics; the shared merged.dta file contains 16,390 estimates, of which 15,917 are RCT rows retained here. The paper-level preregistration coding matches the source article counts: 83 preregistered articles, including 44 with a pre-analysis plan and 39 without one.
Data availability: replication data file merged.dta may be downloaded from https://dataverse.harvard.edu/file.xhtml?fileId=7884702&version=1.0
Data processing: the authors’ reported z-statistics were used directly and the reported coefficients and standard errors were retained. The source zstat field stores unsigned magnitudes, so we apply the sign of the reported coefficient when available. We retain RCT rows and use the source zstat field because the alternative myz field is not populated for all publication years in the replication file. We also retain a grouping variable distinguishing not preregistered studies, preregistered studies without a PAP, and preregistered studies with a PAP.
Model of z-values
| Characteristic | Estimate |
|---|---|
| Probability of significance | 32% |
| Relative probability of publication for |z| < 1.96 | 0.67 |
| Successful replication for |z| > 1.96 | 63% |
| Correct sign for |z| > 1.96 | 98% |
What do these terms mean?
- Probability of significance
- The reported value is the assurance: the proportion of significant results adjusted for publication bias.
- Relative probability of publication
- The relative probability of observing a result below the |z| = 1.96 threshold rather than above it. Values below one indicate lower observation probability below the conventional two-sided significance threshold.
- Successful replication
- The probability that an exact replication has the same sign and |z| greater than 1.96, conditional on the original result having |z| greater than 1.96.
- Correct sign
- The probability that the observed effect has the same direction as the true effect, conditional on an original result with |z| greater than 1.96.
