Lang 2025

Tags:metascience paperrandom sampleprimary outcomeHover over the tag text for details.

Domaineconomics

Data736 papers from econ journals

Note

To download only this data file: Lang.rds (257 KB)

To download all BEAR datasets, click here.

Description

Reference: Lang (2025).

Research question: How frequently are statistically significant rejections of null hypotheses in recent empirical economics false positives?

Data collection: Lang samples empirical articles from the top five economics journals in 2021 and also articles previously collected in Brodeur, Cook, and Heyes, who sampled articles published in 2015 and 2018 in 25 leading economics journals. (See also the “Brodeur” dataset in BEAR for more details.) The sample consists mainly of natural, laboratory, or field experiments, excluding articles focused on theory, structural estimation, or econometric methods.

Lang re-extracted results from all articles rather than using the hypothesis tests extracted by Brodeur, Cook, and Heyes. Research assistants read the abstracts and results text and retained estimates emphasised in the abstract or described by the authors as principal results. When multiple specifications tested the same hypothesis, they retained the authors’ preferred specification or the specification whose magnitude was used in subsequent analysis. Robustness checks and heterogeneity analyses were excluded unless the heterogeneity analysis was itself central to the article.

The source data contain 3,885 extracted hypothesis tests from 736 articles. Lang’s main sample in the cited article restricts these to rejected null hypotheses with \(1.96 \leq |t| < 10\), leaving 2,082 hypothesis tests from 663 articles. The dataset therefore does not identify one focal hypothesis per article. An article can contribute several hypothesis tests. However, only one preferred specification is retained for each hypothesis. (Lang paper also reports analyses restricted to one null hypothesis per article. The supplementary material contains a separate vector with one t-statistic for each of the 736 articles, but this vector does not contain row or article identifiers that allow it to be linked directly to the 3,885-test table. We therefore do not use this in BEAR data.)

Data processing: We use the 3,885 results. We use Lang’s supplied, derounded t-statistic, derounded coefficient (coef_dr in the source data) and standard error (se_dr); the latter two are available in about 95% of cases. (Using either coefficients or t-statistics to calculate z-values leads to practically identical results; we use the latter.) We calculate the corresponding two-sided normal-approximation p-value. We retain some additional columns in Lang.rds.

We also pull DOIs for source articles using bibliographic metadata matching. These DOIs were obtained by us from Crossref, rather than supplied by Lang; the saved assignments are preserved when reprocessing the dataset. This is done automatically via Crossref. The matches are very good in almost all cases. For several edge cases we used LLM review to adjudicate.

Study characteristics:

• Study ID: studyid is Lang_paper_ followed by Lang’s unique_paperid, but with a few minor corrections from us (see GitHub repository for details). Article DOIs are also available as an additional column in Lang.rds.
• Subset: subset distinguishes observations drawn from the Brodeur–Cook–Heyes 2015/2018 source universe from Lang’s 2021 top-five additions.
• Method: The method labels supplied with the Lang data mix study designs, identification strategies, and estimators. Large majority of estimates are one of “RCT”, “DID”, “IV”, “RD”, and “OLS”. Lang.rds retains the original method labels. When constructing BEAR.rds, we collapse some label, e.g. RCT (DID) and RCT (IV) become RCT; DID (matching) becomes DID; and RD (DID) and RD (IV) become RD. For estimates labelled OLS, matching, SYSTEM-GMM, or IV-DID (matching), we retain the estimates but set method to missing.
• The dataset does not have data on type of measure to allow us to classify estimates as in some other datasets in BEAR (e.g. risk risk vs standardised mean difference).

Model of z-values

Characteristic	Estimate
Probability of significance	42%
Relative probability of publication for |z| < 1.96	0.24
Successful replication for |z| > 1.96	70%
Correct sign for |z| > 1.96	99%

What do these terms mean?

Probability of significance

The reported value is the assurance: the proportion of significant results adjusted for publication bias.

Relative probability of publication

The relative probability of observing a result below the |z| = 1.96 threshold rather than above it. Values below one indicate lower observation probability below the conventional two-sided significance threshold.

Successful replication

The probability that an exact replication has the same sign and a |z| greater than 1.96, conditional on the original result having |z| greater than 1.96.

Correct sign

The probability that the observed effect has the same direction as the true effect, conditional on an original result with |z| greater than 1.96.

References

Lang, Kevin. 2025. “How Credible Is the Credibility Revolution?” Journal of Labor Economics 43 (2): 635–63. https://doi.org/10.1086/732772.