Informing Research Choices: Indicators and Judgment

Summary

Lack of a “one-size-fits all” solution to monitoring and measuring research performance is driving a growing international need to develop and implement new methods and, share best practices and lessons learned.

In 2010, the Minister of Industry, on behalf of the Natural Sciences and Engineering Research Council of Canada (NSERC), asked the CCA to examine the international practices and supporting evidence used to assess performance of research in the natural sciences and engineering disciplines.

Informing Research Choices: Indicators and Judgment drew from both international and national approaches for evaluating research performance. Overall, the assessment presents an important opportunity to address science and performance from a uniquely Canadian perspective.

Key Findings

After examining the available evidence, the CCA’s expert panel concluded that quantitative indicators should inform, rather than replace, expert judgment in science assessment for research funding allocation. The Panel also developed four guiding principles to support research funding agencies undertaking science assessment in support of budget allocation, they are: context matters; do not harm; transparency is critical; and expert judgment remains invaluable. The principles are expanded upon with the Panel’s report.

• Many science indicators and assessment approaches are sufficiently robust to be used to assess science performance at the field level in natural sciences and engineering.
• Quantitative indicators should inform, rather than replace, expert judgment in science assessment for research funding allocation.
• International “best practices” offer limited insight into science indicator use and assessment strategies.
• Mapping research funding allocation directly to quantitative indicators is far too simplistic, and is not a realistic strategy.
• There is no compelling reason for certainty that past successes will lead to future successes or past failures to future failures. As a result, science indicators may not always provide a reliable guide to future prospects.