Open Science Collaboration 2015
Description
Reference: Open Science Collaboration (2015).
Research question: replications of a quasi-random sample of 100 experiments in psychology.
Data collection: The authors make a comprehensive dataset of results available with their paper. BEAR includes only the replications because all original studies in that set have \(p < 0.05\).
Data availability: The data file rpp_data.csv is available in the paper’s repository, https://osf.io/ytpuq/overview, under a CC0 1.0 Universal licence.
Data processing: We derived z-values from the replication p-values assuming two-sided p-values (we use the p-value column for the replication, not the p-value for the original study). We applied the sign of the replication correlation, so z-values are signed where that correlation is available. We treated p-values recorded as exactly 0 as truncated and recorded this as a lower bound on z (z-operator >); otherwise we treated p-values as exact (z-operator =). We retained the replication sample size as ss.
In BEAR.rds, the measure is recorded as “correlation”, based on the fact that is the column from original dataset that we use.
Study characteristics:
- BEAR treats the replication studies as RCTs.
- The retained effect measure is the replication correlation.
Model of z-values
| Characteristic | Estimate |
|---|---|
| Probability of significance | 36% |
| Relative probability of publication for |z| < 1.96 | 1.05 |
| Successful replication for |z| > 1.96 | 74% |
| Correct sign for |z| > 1.96 | 98% |
What do these terms mean?
- Probability of significance
- The reported value is the assurance: the proportion of significant results adjusted for publication bias.
- Relative probability of publication
- The relative probability of observing a result below the |z| = 1.96 threshold rather than above it. Values below one indicate lower observation probability below the conventional two-sided significance threshold.
- Successful replication
- The probability that an exact replication has the same sign and a |z| greater than 1.96, conditional on the original result having |z| greater than 1.96.
- Correct sign
- The probability that the observed effect has the same direction as the true effect, conditional on an original result with |z| greater than 1.96.
