Scholars as government-appointed research evaluators: Do they create congruence between their professional quality standards and political demands?

Abstract

All across the globe politically initiated research evaluations are based on "informed peer review" procedures. Scholars are appointed as evaluators and can apply self-defined quality standards in order to overcome shortcomings of standardized measures. Even though there are no binding criteria in these procedures and the quality standards of the scholars' disciplines vary, studies suggest that scholars, in their role as government-appointed research evaluators, assess research uniformly. By drawing on a small-N investigation, this study compares the quality standards scholars apply as government-appointed research evaluators with quality standards they follow as researchers. The study points to a paradox: Criteria scholars refer to while describing the excellence of their own research and criteria they use as evaluators differ and contradict each other. The results are discussed from different angles.