Publication Bias in Meta-Analysis: Detection, Quantification, and Adjustment

A meta-analysis is only as complete as the evidence it can see. If the published literature is a filtered subset of the studies that were actually conducted, the pooled estimate inherits the filter. Publication bias is the systematic error introduced when the availability of study results depends on their findings rather than on their methodological quality. It is the single most consequential threat to the validity of a meta-analysis, because it cannot be repaired by better statistical modeling of the studies that were found — the missing studies are simply absent from the dataset.

Empirical work confirms the problem is not hypothetical. Dwan and colleagues (PLoS One 2008) reviewed empirical studies of selective publication and found that statistically significant results were substantially more likely to be published, submitted, and reported than non-significant results. Outcome reporting bias — where only favorable outcomes within a study are reported — compounds the issue. The consequence is that conventional meta-analyses of published trials tend to overestimate treatment effects and underestimate uncertainty, sometimes to the point of reversing the conclusion.

1. Mechanisms and sources of publication bias

Publication bias operates through several distinct mechanisms, and reviewers should understand which ones plausibly apply to their field. The classic mechanism is selective publication of significant studies: authors are less likely to write up null results, and journals are less likely to accept them. Related mechanisms include time-lag bias (significant results appear faster than null results), language bias (positive findings more often published in English-language journals that dominate indexing), citation bias (significant studies cited more often and therefore more likely to be found by citation chaining), multiple publication bias (the same positive result published repeatedly in overlapping reports), and selective outcome reporting (only favorable endpoints or subgroup analyses reported).

Each mechanism has a different fix. Trial registration and results disclosure address selective publication and outcome reporting; comprehensive multilingual searching and clinical trial registries address language and time-lag bias; careful de-duplication of overlapping reports addresses multiple publication bias. Detection tools can flag the aggregate signature of these mechanisms, but they cannot tell the mechanisms apart. That distinction requires knowledge of the field and the search strategy used.

2. Visual detection: the funnel plot

The funnel plot is the standard graphical tool for detecting publication bias. It plots each study's effect estimate (on the horizontal axis) against a measure of its precision or size (on the vertical axis), usually the standard error, the inverse standard error, or the sample size. Under the assumption that all studies are drawn from the same underlying effect distribution, the plot should resemble a symmetric inverted funnel: small, imprecise studies scatter widely near the bottom, while large, precise studies cluster tightly near the pooled estimate at the top.

When publication bias is present, the bottom of the funnel is lopsided: the small studies that would appear in the lower region are disproportionately those with non-significant or negative results, so the lower part of the funnel shows a visible gap on the side of the null or of harm. Asymmetry in the funnel is the signature of missing small studies. Sterne and Egger (J Clin Epidemiol 2001) showed that the choice of axes matters: plotting the standard error against the log odds ratio (or other scale on which the effect is measured) maximizes the power of the funnel to detect asymmetry, and they provided practical guidance on axis selection.

Contour-enhanced funnel plots (Peters et al., J Clin Epidemiol 2008) shade the regions of the plot corresponding to conventional significance levels (for example p < 0.05 and p < 0.01). If the apparent asymmetry is caused by publication bias, most of the missing studies should lie in the non-significant region; if the asymmetry persists across the entire funnel, it may instead reflect true heterogeneity or differences in study quality. This visual aid helps reviewers distinguish publication bias from other causes of asymmetry before applying any statistical test.

3. Statistical tests for small-study effects

Formal tests quantify the asymmetry that the funnel plot displays. The most widely used is Egger's regression test (Egger et al., BMJ 1997), which regresses the standardized effect (effect divided by its standard error) on precision (inverse standard error). Under symmetry, the regression line passes through the origin; a non-zero intercept indicates small-study effects. Egger's test has good power when genuine small-study bias exists but is sensitive to the presence of a few influential small studies.

Alternatives and refinements address specific settings. Begg and Mazumdar's rank correlation test (Biometrics 1994) assesses the correlation between effect sizes and their variances using Kendall's tau; it has lower power than Egger's test but is more robust to outliers. For binary outcomes, the Harbord modified test (Stat Med 2006), the Peters test (JAMA 2006), and the arcsine test of Rücker and colleagues (Stat Med 2008) avoid the artifactual correlation between the log odds ratio and its variance in sparse data, which can inflate false-positive rates of the original Egger test. The Doi plot with the LFK index (Furuya-Kanamori et al., Int J Evid Based Healthc 2018) offers an alternative graphical and quantitative asymmetry assessment based on a different effect-size transformation.

No test is a definitive diagnosis. Each test examines small-study effects, which are a necessary but not sufficient condition for publication bias: small studies may differ from large studies in quality, intervention intensity, or population (true heterogeneity), producing asymmetry without any suppression of results. Tests are also underpowered when the number of studies is small. Sterne and colleagues (BMJ 2011), summarizing the Cochrane and PRISMA guidance, recommended that formal tests of funnel-plot asymmetry generally be used only when the meta-analysis includes at least ten studies, and that results be interpreted cautiously with fewer studies.

4. Interpreting asymmetry without overclaiming

The greatest risk in publication-bias analysis is over-interpretation. Funnel-plot asymmetry is compatible with at least four explanations: publication bias, true between-study heterogeneity in effects, poor methodological quality of smaller studies, and chance. The reviewer's job is to weigh these explanations, not to assume the first one.

Practical heuristics help. If the asymmetry disappears after excluding the smallest and most imprecise studies, the result is sensitive to small-study effects. If a contour-enhanced funnel shows the missing studies concentrated in the non-significant region, publication bias is plausible. If heterogeneity measures (I², tau²) are high and asymmetry persists across all regions of the funnel, true differences between small and large studies should be investigated through meta-regression on design features, risk of bias, or population characteristics. Reporting the tests, the number of studies, and the alternative explanations in the text is more informative than a single p-value from one test.

When only a handful of studies exist, formal testing is not merely underpowered — it can be actively misleading, because with very few studies the distribution of the test statistic under the null is poorly approximated. Reviewers should present the funnel plot descriptively, state the limitation, and rely on the search strategy and registry-based gap analysis to argue about completeness of the evidence rather than on a low-power significance test.

5. Adjustment methods: estimating the corrected effect

When publication bias is judged plausible, adjustment methods estimate what the pooled effect would be if the missing studies were present. The most widely reported is the trim-and-fill method of Duval and Tweedie (Biometrics 2000). Trim-and-fill iteratively removes (trims) the most extreme studies on the asymmetric side of the funnel, recomputes the pooled estimate, imputes (fills) the mirror-image studies that would restore symmetry, and recomputes the adjusted estimate. Its output — an adjusted effect and confidence interval — is intuitive and available in most meta-analysis packages, which explains its popularity.

Trim-and-fill has important limitations. It assumes that the asymmetry is caused by publication bias and that the missing studies are mirror images of the trimmed studies, an assumption that may not hold when asymmetry reflects true heterogeneity. It can also perform poorly when fewer than half the studies are on the dominant side, and it does not model the mechanism by which studies were suppressed. For these reasons, the Cochrane Handbook treats trim-and-fill as one sensitivity analysis among several, not as a definitive correction.

Selection models are more principled. The Copas selection model (Copas and Shi, Stat Methods Med Res 2001) explicitly models the probability that a study is published as a function of its result, estimating both the pooled effect and the publication probability parameters in a joint model, with sensitivity analysis over the assumed selection strength. p-curve analysis (Simonsohn et al., Perspect Psychol Sci 2014) takes a different approach: it examines the distribution of p-values among significant results only, exploiting the fact that p-values of genuine effects are right-skewed toward small values, while selective reporting of just-significant results produces a left-skewed or flat distribution. p-curve estimates the true effect and detects p-hacking, but it is designed for sets of studies with a common effect and is less suited to heterogeneous clinical meta-analyses.

Adjusted estimates should always be presented alongside the conventional estimate, never instead of it, and the choice of adjustment method should be prespecified or justified. A large discrepancy between the conventional and adjusted estimates is a red flag: the evidence base may be too fragile to support clinical recommendations.

6. GRADE, risk-of-bias assessment, and reporting

GRADE rates the certainty of evidence across five domains, and publication bias is one of them (Guyatt et al., J Clin Epidemiol 2011). Reviewers should consider downgrading the certainty of evidence when publication bias is strongly suspected: for example, when studies are small and predominantly industry-funded, when the evidence base is limited to published trials with no registry cross-check, or when funnel-plot asymmetry is pronounced and unexplained. The GRADE approach emphasizes that publication bias should be suspected even when it cannot be proven, especially for meta-analyses of small studies with few events.

Reporting guidance now requires explicit treatment of missing results. The PRISMA 2020 statement (Page et al., BMJ 2021) asks authors to describe any methods used to assess the risk of bias due to missing results in a synthesis, to state which assessments were performed, and to report the results. The Cochrane Handbook (version 6.x) devotes a full chapter to assessing the risk of bias due to missing results, recommending a combination of comprehensive search, trial-registry comparison, funnel-plot examination, and formal tests, each reported transparently with the number of studies involved.

Completeness of the search is the first line of defense. Searches limited to bibliographic databases and English-language journals systematically miss unpublished and non-English studies. Supplementing database searches with trial registries (such as ClinicalTrials.gov and the WHO ICTRP), preprint servers, conference abstracts, and reference lists reduces — though never eliminates — the gap between conducted and retrieved studies. Reviewers should quantify the gap when possible by comparing included trials against registry records.

7. Prevention: making bias harder

The most effective response to publication bias is preventive rather than corrective. Prospective trial registration creates a public record of planned trials, enabling reviewers to identify missing results and enabling regulators to track non-publication. The AllTrials initiative and related campaigns have pushed for registration of all interventional studies and reporting of all results. Systematic review protocols registered in PROSPERO document the planned search and synthesis methods before results are known, protecting against post hoc analytical flexibility.

For individual researchers, the practical steps are to register protocols, to pre-specify the publication-bias assessment plan (which funnel plot, which tests, which adjustment methods, and the minimum number of studies for formal testing), and to commit to publishing results regardless of direction. Journal policies that encourage or mandate result reporting, and repositories that accept null results, shift the incentive structure that produces publication bias in the first place.

8. Evidence summary table

9. Actionable Steps: Run a Publication-Bias Assessment

10. Common mistakes in publication-bias assessment

The most common mistake is running a single test and declaring the result: testing asymmetry in a meta-analysis of five studies, or applying Egger's test to sparse binary data without an appropriate modification, produces conclusions that are not supported by the method's assumptions. A second common error is treating trim-and-fill output as the corrected truth. Trim-and-fill imputes studies under a symmetry assumption that fails when asymmetry reflects true heterogeneity; presenting its adjusted estimate as the definitive effect can mislead readers. A third error is diagnosing publication bias from asymmetry alone while ignoring the field-specific search strategy and registry evidence — asymmetry has multiple causes, and the search design determines which causes are plausible. Finally, many reviews omit the publication-bias assessment entirely, or bury it in a supplementary file, even though PRISMA 2020 explicitly asks authors to report how they assessed the risk of bias due to missing results.

11. Relation to other meta-analysis methods

Publication bias is one member of a family of threats to meta-analytic validity that this series has covered. Trial Sequential Analysis (TSA) addresses random error from repeated significance testing in cumulative meta-analysis; publication-bias assessment addresses systematic error from missing results. GRADE provides the overall certainty framework that integrates both: TSA informs the imprecision domain, while publication-bias assessment informs its own domain. Heterogeneity assessment and meta-regression (covered in earlier posts) explain funnel asymmetry that is not caused by suppression. A complete evidence-synthesis workflow therefore combines comprehensive searching, heterogeneity analysis, TSA for repeated-testing control, and a transparent publication-bias assessment, each reported separately and integrated in the final certainty rating.

Conclusion

Publication bias is not a statistical nuisance that better models can quietly fix; it is a property of how evidence becomes visible, and it must be addressed at the level of search design, assessment, and reporting. A rigorous publication-bias workflow combines comprehensive searching with registry cross-checks, visual funnel-plot examination with contour enhancement, formal small-study-effect tests used within their assumptions, adjustment methods reported as sensitivity analyses, and an explicit GRADE rating. No single test or plot proves the absence of missing studies, and no adjustment method fully reconstructs what was never published. What a transparent assessment can do is show readers how fragile the evidence is, so that clinical and policy conclusions are drawn with appropriate caution.