More on Improving Reviewing Quality with Double-Blind Reviewing, External Review Committees, Author Response, and in Person Program Committee Meetings

  • Kathryn S McKinley

Publication

Peer review is the central mechanism that scientists use to evaluate research manuscripts, artifacts, and individual researchers, and thus advance the state of scientific knowledge and determine career trajectories. Peer reviewing seeks to evaluate research using qualified experts, who have the scientific training, experience, and skills to judge the merit of the work. Just as scientific progress and methods are evolving, so is the peer review process. Conferences are the primary publication venues for computer scientists. Current challenges for peer reviewing in computer science include (1) obtaining expert reviews as the breadth and depth of our field grows; (2) accommodating growth in the amount of research, number of researchers, and submissions; and (3) incorporating best practices based on social science research on nepotism and bias.

This article proposes the following best practices in light of these challenges: (1) double-blind until accept to further reduce bias and enable rejected submission to remain blind in subsequent submissions; (2) the need for external review committees (versus ad hoc assignments) to handle increasing numbers and breadth of submissions, ease the implementation of double-blind reviewing, and improve overall review quality with more expert reviews; (3) the use of author response to improve reviewer accountability; and (4) in person program committee meetings to communicate, establish, and constantly apply scientific standards and build community. While (2) and (3) are now common, (1) double-blind implementation remains varied, and I believe (4) is undervalued as a mechanism for improving community and reviewing accountability (trust and verify).