{"pk":18432,"title":"Development and Validation of a Scoring Rubric for Editorial Evaluation of Peer-review Quality: A Pilot Study","subtitle":null,"abstract":"<b>Introduction: </b>Despite the importance of peer review to publications, there is no generally accepted approach for editorial evaluation of a peer review’s value to a journal editor’s decision-making. The graduate medical education editors of the Western Journal of Emergency Medicine Special Issue in Educational Research &amp; Practice (Special Issue) developed and studied the holistic editor’s scoring rubric (HESR) with the objective of assessing the quality of a review and an emphasis on the degree to which it informs a holistic appreciation for the submission under consideration.<p></p><p><b><b>Methods: </b></b>Using peer-review guidelines from several journals, the Special Issue’s editors formulated the rubric as descriptions of peer reviews of varying degree of quality from the ideal to the unacceptable. Once a review was assessed by each editor using the rubric, the score was submitted to a third party for blinding purposes. We compared the performance of the new rubric to a previously used semantic differential scale instrument. Kane’s validity framework guided the evaluation of the new scoring rubric around three basic assumptions: improved distribution of scores; relative consistency rather than absolute inter-rater reliability across editors; and statistical evidence that editors valued peer reviews that contributed most to their decision-making.</p><p></p><p><b><b><b>Results: </b></b></b>Ninety peer reviews were the subject of this study, all were assessed by two editors. Compared to the highly skewed distribution of the prior rating scale, the distribution of the new scoring rubric was bell shaped and demonstrated full use of the rubric scale. Absolute agreement between editors was low to moderate, while relative consistency between editor’s rubric ratings was high. Finally, we showed that recommendations of higher rated peer reviews were more likely to concur with the editor’s formal decision.</p><p></p><p><b><b><b><b>Conclusion:</b></b></b></b> Early evidence regarding the HESR supports the use of this instrument in determining the quality of peer reviews as well as its relative importance in informing editorial decision-making.</p>","language":null,"license":{"name":"Creative Commons Attribution 4.0","short_name":"CC BY 4.0","text":"Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.\r\n\r\nNo additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.","url":"https://creativecommons.org/licenses/by/4.0"},"keywords":[{"word":"peer-review"},{"word":"Editorial Assessment"},{"word":"validity"}],"section":"Research Methods","is_remote":true,"remote_url":"https://escholarship.org/uc/item/98q1j29m","frozenauthors":[{"first_name":"Jeffrey","middle_name":"N.","last_name":"Love","name_suffix":"","institution":"Georgetown University School of Medicine, Department of Emergency Medicine, Washington, District of Columbia","department":"None"},{"first_name":"Anne","middle_name":"M.","last_name":"Messman","name_suffix":"","institution":"Wayne State University, Department of Emergency Medicine, Detroit, Michigan","department":"None"},{"first_name":"Jonathan","middle_name":"S.","last_name":"Ilgen","name_suffix":"","institution":"University of Washington, Department of Emergency Medicine, Seattle, Washington","department":"None"},{"first_name":"Chris","middle_name":"","last_name":"Merritt","name_suffix":"","institution":"Alpert Medical School of Brown University, Department of Emergency Medicine, Providence, Rhode Island; Alpert Medical School of Brown University, Department of Pediatrics & Emergency Medicine, Providence, Rhode Island","department":"None"},{"first_name":"Wendy","middle_name":"C.","last_name":"Coates","name_suffix":"","institution":"David Geffen School of Medicine at UCLA, Department of Emergency Medicine, Los Angeles, California","department":"None"},{"first_name":"Douglas","middle_name":"S.","last_name":"Ander","name_suffix":"","institution":"Emory University, Department of Emergency Medicine, Atlanta, Georgia","department":"None"},{"first_name":"David","middle_name":"P.","last_name":"Way","name_suffix":"","institution":"Ohio State University College of Medicine, Department of Emergency Medicine, Columbus, Ohio","department":"None"}],"date_submitted":"2023-08-16T23:13:55+03:00","date_accepted":"2023-08-16T23:13:55+03:00","date_published":"2024-02-28T16:00:00+02:00","render_galley":{"label":"Final Article","type":"pdf","path":"https://journalpub.escholarship.org/westjem/article/18432/galley/10081/download/"},"galleys":[{"label":"Final Article","type":"pdf","path":"https://journalpub.escholarship.org/westjem/article/18432/galley/10081/download/"}]}