Open Science Collaboration 2015
Description
Reference: Open Science Collaboration (2015).
Research question: replications of a quasi-random sample of 100 experiments in psychology.
Data collection: The authors make a comprehensive dataset of results available with their paper. BEAR includes only the replications because all original studies in that set have \(p < 0.05\).
Data availability: The data file rpp_data.csv is available in the paper’s repository, https://osf.io/ytpuq/overview, under a CC0 1.0 Universal licence.
Data processing: We derived z-values from the replication p-values assuming two-sided p-values (we use the p-value column for the replication, not the p-value for the original study). We applied the sign of the replication correlation, so z-values are signed where that correlation is available. We treated p-values recorded as exactly 0 as truncated and recorded this as a lower bound on z (z-operator >); otherwise we treated p-values as exact (z-operator =). We retained the replication sample size as ss.
doi_replication identifies the Open Science Collaboration paper reporting these replication results; it is shared across studies.
In BEAR.rds, the measure is recorded as “correlation”, based on the fact that is the column from original dataset that we use.
Study characteristics:
- BEAR treats the replication studies as RCTs.
- The retained effect measure is the replication correlation.
Model of z-values
| Characteristic | Estimate |
|---|---|
| Probability of significance | 36% |
| Relative probability of publication for |z| < 1.96 | 1.05 |
| Successful replication for |z| > 1.96 | 74% |
| Correct sign for |z| > 1.96 | 98% |
What do these terms mean?
- Probability of significance
- The reported value is the assurance: the proportion of significant results adjusted for publication bias.
- Relative probability of publication
- The relative probability of observing a result below the |z| = 1.96 threshold rather than above it. Values below one indicate lower observation probability below the conventional two-sided significance threshold.
- Successful replication
- The probability that an exact replication has the same sign and a |z| greater than 1.96, conditional on the original result having |z| greater than 1.96.
- Correct sign
- The probability that the observed effect has the same direction as the true effect, conditional on an original result with |z| greater than 1.96.
