Agreement between ChatGPT and human-derived multilevel meta-analyses: a reproducibility study across clinical evidence syntheses

Resumo original em inglês

O que aconteceu

Large language models are increasingly integrated into biomedical research workflows, yet their ability to reliably reproduce complex statistical analyses remains insufficiently evaluated. While prior studies suggest that ChatGPT can replicate conventional meta-analyses, its performance in recalculating statistically demanding multilevel meta-analyses is unknown.

We conducted a systematic review to identify published multilevel meta-analyses (i.e., models accounting for dependency among multiple effect sizes within studies) of hip-related clinical studies that provided extraction tables enabling independent recalculation. For each outcome, pooled estimates and heterogeneity parameters were recomputed using ChatGPT under predefined analytical assumptions and compared with statistician-derived results. Agreement was assessed using absolute and relative differences and Bland-Altman analyses for pooled effects, and qualitative comparison of heterogeneity measures (I², τ²) based on absolute differences.

O que encontraram

Seven studies, each reporting a multilevel meta-analysis, comprising more than 40 pooled effect estimates were included. ChatGPT-derived pooled estimates showed close numerical agreement with human-derived results, with small median absolute differences for continuous (2.25) and binary outcomes (0.005) and minimal overall bias. Heterogeneity estimates demonstrated similar concordance, with predominantly small differences that did not alter qualitative interpretation. Larger deviations were limited to a small number of outcomes with extreme heterogeneity or inconsistent input structures.

Por que isso importa

ChatGPT can reproducibly approximate the results of multilevel meta-analyses under predefined analytical conditions. These findings support the potential role of large language models as adjunct tools for reproducibility assessment and methodological validation in evidence synthesis, while emphasizing the continued need for expert oversight.

Resumo publicado sob licença CC BY. A tradução para o português ainda não chegou a este estudo.