a reliability-generalization study of journal peer reviews a multilevel meta-analysis of inter-rater reliability and its determinantsreliability-generalization研究杂志同行评审的多层次分析两分的可靠性及其决定因素.pdfVIP

A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants 1 ¨ 2 2,3 Lutz Bornmann *, Rudiger Mutz , Hans-Dieter Daniel 1 Max Planck Society, Munich, Germany, 2 Professorship for Social Psychology and Research on Higher Education, ETH Zurich, Zurich, Switzerland, 3 Evaluation Office, University of Zurich, Zurich, Switzerland Abstract Background: This paper presents the first meta-analysis for the inter-rater reliability (IRR) of journal peer reviews. IRR is defined as the extent to which two or more independent reviews of the same scientific document agree. Methodology/Principal Findings: Altogether, 70 reliability coefficients (Cohen’s Kappa, intra-class correlation [ICC], and Pearson product-moment correlation [r]) from 48 studies were taken into account in the meta-analysis. The studies were based on a total of 19,443 manuscripts; on average, each study had a sample size of 311 manuscripts (minimum: 28, maximum: 1983). The results of the meta-analysis confirmed the findings of the narrative literature reviews published to date: The level of IRR (mean ICC/r2 = .34, mean Cohen’s Kappa = .17) was low. To explain the study-to-study variation of the IRR coefficients, meta-regression analyses were calculated using seven covariates. Two covariates that emerged in the meta- regression analyses as statistically significant to gain an approximate homogeneity of the intra-class correlations indicated that, firstly, the more manuscripts that a study is based on, the smaller the reported IRR coefficients are. Secondly, if the information of the rating system for reviewers was reported in a study, then this was associated with a smaller IRR coefficient than if