Thank you to the 500+ Experts Who Have Peer Reviewed CCA Reports

“What do we want? Evidence-based change! When do we want it? After peer review!”

–         Anonymous sign photographed at March for Science

Thirty-five years ago I received a phone call from University of Montreal professor David Roy inviting me to join a grant application he was submitting to a Canadian federal funding agency, the goal of which was to develop criteria for the evaluation of multidisciplinary bioethics research. It was an intriguing project: unlike the criteria for assessing single discipline studies, where the methods and experts to assess them were established, bioethics involved philosophers, lawyers, psychologists, theologians and countless others, each of whom had discipline-specific approaches to excellence, scientific merit, and impact ― but no two of which were identical. The project was not funded; perhaps it was not a sufficiently meritorious proposal compared to other submissions, but it may have been that peer reviewers lacked the criteria to assess the proposal. It stuck with me that while peer review remains one of the most important enablers for disseminating quality science ever invented, it too has its own challenges, and opportunities for improvement.

My story is a little more than a minor footnote in the long history of peer review, a practice first used in 1731 by the Royal Society of Edinburgh for a collection of medical articles in Medical Essays and Observations. Indeed, peer review has evolved considerably since its early 18th century introduction, varying by degree and use: the British Medical Journal started using it in 1893, but the equally prestigious Lancet did not use peer review until 1976. By the mid-to-late 20th century most major journals (and granting agencies) adopted a process which we recognize today: recognized experts who are unbiased and conflict free review the content of a manuscript or grant application, and offer an objective assessment of its merits and methods that is sufficiently detailed to serve the purpose of giving editors and funders a way to choose which papers to publish and which projects to support.

Peer review has not stood still. Online submissions were not always the norm. A close friend once relayed a story of being so worried that his grant application would not arrive at a granting agency by regular three-day mail to be “date stamped” that he booked a flight, and flew to the city where the agency was located. (Also, he took no luggage other than a box filled with the requisite number of photocopies required for peer review.)

The gold standard of using single and double blinded reviews has been the norm, though occasionally there are experiments where submitter and reviewer meet face-to-face.

CCA’s Peer Review Process – Not the Academic Model

Which brings us to the peer review process used by the Council of Canadian Academies. The CCA focuses its efforts on carrying out assessments of the state of knowledge on topics of policy importance to Canada and Canadians. Over 13 years we’ve carried out more than 45 assessments using expert panels who review what is known about a topic, what isn’t known, and how to evaluate the meaning of both types of findings. But our process would not be complete, let alone of value to the sponsors who request assessments of us, were it not for the rigorous peer review process we use. Unlike the journal/grant process however, the CCA uses its peer reviewers in a slightly different way.

• For every panel report, there will be an almost identical number of peer reviewers (an expert panel of 12 to 15 members will have about 12 to 15 peer reviewers.)
• Peer reviewers provide their input prior to the final completed report, to allow the panel to consider the comments and adopt where necessary to improve the overall report.
• Each peer review written comment (which can number in the hundreds) is considered at a CCA panel meeting.
• Peer reviewers are not asked to recommend whether a CCA report should be published as is, published with minor (or major) revisions, or rejected for publication.
• A “Peer Review Monitor” selected from CCA’s Scientific Advisory Committee must attest to the CCA Board of Directors whether the peer review process was followed.

If it takes a village to raise a child, it takes a metropolis to peer review a CCA report. Since 2005, more than 500 individuals have provided reviews for our work.  It is work they take seriously, and have improved our reports in untold ways. I continue to be both amazed by and indebted to these individuals, who do this work uncompensated because they care about the quality of the science being reported. Perhaps this is why the eminent authority on scientific integrity, Francis Macrina has described peers and peer reviewers as “sentinels on the road of scientific discovery and publication.”

Regardless of whether you have reviewed others’ work for journals, grant applications, or other avenues of dissemination, thank you for your contributions.

Eric M. Meslin, PhD, FCAHS
President and CEO, Council of Canadian Academies