Documentation

Standard procedure for dealing with p-values and confidence intervals

Most datasets in BEAR come with pre-computed effect sizes and standard errors. Some, especially databases of clinical trials and large sets of scraped studies, report p-values. Moreover, reporting in clinical trial databases is done by investigators, and the data-entry fields are not standardised, requiring more extensive clean-up and a procedure for deriving z-values. Lastly, in some cases datasets contain confidence intervals and effect sizes, but z- and p-values may be missing.

Data cleaning and default values

For confidence interval levels, we set implausible values to missing (e.g. values above 100 or below 0) and standardise proportions and percentages: investigators sometimes report 90 and sometimes 0.9 for a 90% confidence interval. If a confidence interval is present but its level is missing, we assume 95% by default. Some data state whether CIs or p-values are one- or two-sided.

Similarly, when working with p-values we assume a two-sided test when no additional information is stated.

Confidence intervals

When working with CIs, we analyse ratio measures on the log scale when the lower bound, upper bound, and point estimate are all strictly positive; we keep non-ratio measures on the raw scale. When constructing BEAR.rds we drop rows where where the lower bound is higher than the upper bound.

To derive SEs from confidence intervals, we first compute the critical value. To avoid undefined critical values (e.g. at 100% or extremely close to it), we bound the implied \(\alpha\) at a small value. After conversion to the chosen scale, we compute the two half-widths around the reported point estimate, convert each to a standard error, and use their average when both are finite.

Where additional information is available, we look for potentially non-Wald intervals by matching text in the parameter-type column. We search for keywords such as median, Hodges, posterior/Bayes, exact, Fieller, bootstrap, and permutation. For flagged estimates, we prefer to derive the z-value from the p-value.

Moreover, we compute a simple symmetry diagnostic for the confidence interval on the chosen scale, defined as the smaller of the two half-widths divided by the larger. We treat intervals as “symmetric enough” only if this ratio exceeds 0.8. Otherwise, we use a p-derived z-statistic where one is available; if not, we drop the row from BEAR.rds. The original values remain available in the individual datasets for other investigations.

In the current version, this rule either selects p-values for about 5–10% of eligible clinical-trial registry rows or removes CI-only rows, including some Barnett–Wren intervals. A 0.95 cut-off would remove or replace roughly twice as many registry intervals. The implications depend on the analysis.

p-values

When a source reports t-statistics and degrees of freedom, we first derive the two-sided p-value from the t distribution and then derive the z-value (retaining the sign of the original t-statistic, if available).

For clinical trials databases (ClinicalTrials.gov and EU CTR), we sometimes find ambiguity in how p-values are recorded. Some significant results report p-values close to 1, suggesting that authors report the cumulative distribution, \(\Phi(z)\). Therefore, for each row we also calculate \(2\cdot\min\{p,1-p\}\). When a CI-derived z-value is available, we choose the p-value that gives the closest z-value.

To avoid dropping rows, we retain unsigned z-statistics when the sign cannot be determined. If the sign of the effect is known, we apply it to the p-derived z; otherwise, we use its absolute value.

Working with correlations

In literature where correlations are typically reported as outcomes (e.g. psychology) we find mixtures of Fisher-transformed and “raw” correlations, typically Pearson \(r\).

As a reminder, for Pearson correlation \(r\) with a verified sample size \(n>3\), Fisher’s transformation is

\[ f = \operatorname{atanh}(r),\qquad \mathrm{SE}(f) = \frac{1}{\sqrt{n-3}}.\qquad \] (We use \(f\) notation since \(z\) is prominently used in BEAR for z-values.) If a sample size is unavailable but the standard error for “raw” correlation is supplied, we can approximate the transformed standard error by \(\mathrm{SE}(f)\approx\mathrm{SE}(r)/(1-r^2)\) for \(|r|<1\).

However, in BEAR we use the scale that was chosen by authors. We use the “effect_scale” column in BEAR to differentiate “fisher_z” and “raw” correlations.

In some data we found mixing of SE’s and point estimates on different scales; in that case we did not include these observations in BEAR.rds. Note as well large discrepancies in z-values depending on whether Fisher’s transformation is used or not and correlation is high. For that reason in one dataset (Sladekova et al) we only retain Fisher-transformed correlations.

Choice of a single z-value when multiple candidates are available

In situations where \(z\) can be derived from both CIs and p-values, we choose a single “best” z-statistic per row as follows. We prefer the CI-derived \(z\) when it looks like a plausible Wald interval on the chosen scale, meaning: (1) finite \(z\) and standard error, (2) positive standard error, (3) no non-Wald flag from a keyword search of the measure label, and (4) either missing symmetry information or symmetry above 0.8. Otherwise, we use the p-derived z.

Truncation and infinite z-values

For reported two-sided p-value bounds, \(p<c\) gives a lower bound on \(|z|\). BEAR records this inequality using z_operator. The operator applies to the magnitude, including when the stored z-value is negative. For example, a stored value of \(-1.96\) with operator > means a negative direction and magnitude greater than 1.96, not \(z>-1.96\).

Exact zero p-values can produce infinite z-values. We retain infinities in BEAR source data, although signal-to-noise models truncate these infinite values.