Brodeur et al. 2024
To download only this data file: Brodeur.rds (819 KB)
To download all BEAR datasets, click here.
Description
Reference: Brodeur et al. (2024).
Research question: are preregistration and pre-analysis plans associated with reduced p-hacking and publication bias in economics?
Data collection: The authors manually extracted test statistics from articles identified as RCT studies in leading economics journals published between 2018 and 2021. Extraction was performed by researchers. They collected coefficients of interest from results tables, excluding constants, balance checks, robustness checks, regression controls, and placebo tests. Each row of data corresponds to a single reported test statistic, linked to a paper-level identifier. Unlike most other sources in BEAR, this means the source data have dozens of estimates per paper. The paper reports 314 articles and 15,992 test statistics; the shared merged.dta file contains 16,390 estimates, of which 15,917 are rows from RCTs included here. The paper-level preregistration coding matches the counts reported in the article: 83 preregistered articles, including 44 with a pre-analysis plan and 39 without one.
Data availability: The replication data file merged.dta may be downloaded from https://dataverse.harvard.edu/file.xhtml?fileId=7884702&version=1.0.
Data processing: The authors’ reported z-statistics were used directly, but the reported coefficients and standard errors are also available. The source zstat field stores unsigned magnitudes, so we apply the sign of the reported coefficient when available. (We use RCT rows and the source zstat field because the alternative myz field in original data is not populated for all publication years in the replication file.)
We obtained article DOIs by matching titles, authors, journals and publication years against Crossref metadata. The additional doi column contains the highest-ranked candidate; uncertain matches are flagged for review. These article identifiers are distinct from the trial-registration DOIs supplied with the source data. They do not replace the paper titles used as study IDs.
Study characteristics: - Study ID: we use the paper title; we include doi as an additional column in Brodeur.rds - subset records preregistration and pre-analysis-plan status, which was the focus of the original study. - All estimates come from RCTs. - There is no information on effect-size measure (e.g. SMD vs ratio). - subset records preregistration and pre-analysis-plan status.
Model of z-values
| Characteristic | Estimate |
|---|---|
| Probability of significance | 32% |
| Relative probability of publication for |z| < 1.96 | 0.67 |
| Successful replication for |z| > 1.96 | 63% |
| Correct sign for |z| > 1.96 | 98% |
What do these terms mean?
- Probability of significance
- The reported value is the assurance: the proportion of significant results adjusted for publication bias.
- Relative probability of publication
- The relative probability of observing a result below the |z| = 1.96 threshold rather than above it. Values below one indicate lower observation probability below the conventional two-sided significance threshold.
- Successful replication
- The probability that an exact replication has the same sign and a |z| greater than 1.96, conditional on the original result having |z| greater than 1.96.
- Correct sign
- The probability that the observed effect has the same direction as the true effect, conditional on an original result with |z| greater than 1.96.
