Costello and Fox 2022; Yang et al 2023, 2024
To download only this data file: CostelloFox.rds (2.3 MB); Yang.rds (255 KB)
To download all BEAR datasets, click here.
Description
Reference: These two datasets come from the same replication package and were processed using the shared pipeline in Yang et al. 2024 (Yang et al. 2024). The original references for the datasets are Yang et al. 2023 and Costello and Fox 2022 (Yang et al. 2023; Costello and Fox 2022).
Research question: decline effects in ecology (Costello and Fox) and publication bias and performance of empirical research (Yang et al)
Data collection: Costello and Fox examined 466 meta-analyses identified by searching Web of Science. The reproducibility dataset includes 88,000 rows across 466 analyses in 232 papers. Subsequently, Yang et al. removed duplicated studies and zeroes, discarding about 20% of rows. Yang et al. collected 102 meta-analyses published in relevant journals between 2010 and 2019, after removing records duplicated in the Costello and Fox dataset.
Data availability: The two datasets may be downloaded from https://github.com/Yefeng0920/replication_EcoEvo_git/. They are available under a CC BY 4.0 licence.
Data processing: The only additional processing was to clean up study years. Signed z-values were computed as effect sizes divided by their standard errors. For Yang et al, BEAR uses the source file names as meta-analysis identifiers.
Costello and Fox:
In BEAR.rds, we re-label the effect measures as SMD, response ratio and correlation. The individual dataset keeps the original Yang et al. coding (Zr, lnRR, SMD).
Yang:
In BEAR.rds, we code Fisher’s z as “correlation” and lnRR as “response_ratio” for consistency with other datasets.
Study characteristics:
Study ID: for Costello and Fox, we use
study2; for Yang, we usestudy_ID.Meta-analysis ID: for Costello and Fox, we use
meta.analysis.id; for Yang, we usemeta_id.Topic: for Costello and Fox, we use ecology, evolution or other ecology/evolution. Yang has no topic classification.
Costello and Fox: the primary-study design is not classified.
Costello and Fox: effects are standardised mean differences, log response ratios, Fisher’s z correlations, or less common measures. Less common measures include regression coefficients, log odds ratios and mean differences. Some standardised mean differences are recorded as absolute values.
Yang: the primary-study design is not classified.
Yang: effects are standardised mean differences, log response ratios, or Fisher’s z correlations.
Model of z-values
This documentation page covers more than one fitted dataset, so the fitted models are shown separately.
| Characteristic | Estimate |
|---|---|
| Probability of significance | 39% |
| Relative probability of publication for |z| < 1.96 | 0.74 |
| Successful replication for |z| > 1.96 | 74% |
| Correct sign for |z| > 1.96 | 98% |
| Characteristic | Estimate |
|---|---|
| Probability of significance | 37% |
| Relative probability of publication for |z| < 1.96 | 0.73 |
| Successful replication for |z| > 1.96 | 73% |
| Correct sign for |z| > 1.96 | 98% |
What do these terms mean?
- Probability of significance
- The reported value is the assurance: the proportion of significant results adjusted for publication bias.
- Relative probability of publication
- The relative probability of observing a result below the |z| = 1.96 threshold rather than above it. Values below one indicate lower observation probability below the conventional two-sided significance threshold.
- Successful replication
- The probability that an exact replication has the same sign and a |z| greater than 1.96, conditional on the original result having |z| greater than 1.96.
- Correct sign
- The probability that the observed effect has the same direction as the true effect, conditional on an original result with |z| greater than 1.96.
Costello and Fox: ecology & evolution
Yang et al: ecology & evolution
