Szucs and Ioannidis 2017
Description
Reference: Szucs and Ioannidis (2017).
Research question: empirical assessment of published effect sizes, statistical significance, and power in the literature on cognitive neuroscience and psychology.
Data availability: The article and supporting data are available from PLOS Biology at https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.2000797.
Data description and source: The source contains 26,841 text-mined t-test records from 3,801 papers in 18 journals (cognitive neuroscience, psychology, and medical). The processed dataset preserves all three fields; in the main BEAR dataset, we include only the cognitive neuroscience subset (since the psychology and medical literature are well covered by other datasets in BEAR).
Notes: The extraction targeted t-test records reported in article texts and did not mine tables.
Data processing: We followed our default procedure for deriving z-values from t-values.
Study characteristics:
- Study ID: to make look-ups of articles easier, we use
journal_<journal_code>_article_<article_id>. - Topic: we use the source
fieldclassification; note thatBEAR.rdscontains only cognitive neuroscience rows. - The source data does not classify study designs.
- The original dataset labels all results as coming from a t-test, but there is no additional information on what was the underlying measure.
Model of z-values
| Characteristic | Estimate |
|---|---|
| Probability of significance | 35% |
| Relative probability of publication for |z| < 1.96 | 0.35 |
| Successful replication for |z| > 1.96 | 62% |
| Correct sign for |z| > 1.96 | 99% |
What do these terms mean?
- Probability of significance
- The reported value is the assurance: the proportion of significant results adjusted for publication bias.
- Relative probability of publication
- The relative probability of observing a result below the |z| = 1.96 threshold rather than above it. Values below one indicate lower observation probability below the conventional two-sided significance threshold.
- Successful replication
- The probability that an exact replication has the same sign and a |z| greater than 1.96, conditional on the original result having |z| greater than 1.96.
- Correct sign
- The probability that the observed effect has the same direction as the true effect, conditional on an original result with |z| greater than 1.96.
