Peer review and knowledge dynamics Terttu Luukkonen Research Institute of the Finnish Economy ERC Workshop Monitoring the performance and quality of peer review systems 28-29 November, 2013, Brussels Focus • Evaluation of research proposals • Groundbreaking/pathbreaking/frontier/ transformative research Content • Defining groundbreaking research – potential differences between fields in the understanding • Phases in the emergence and growth of new research areas • Comparison of peer review with other methods in its ability to select groundbreaking research • Conclusions Categories of groundbreaking • Discovery of a novel phenomenon (serendipitous discoveries and others) • New method or technique or their combination as an enabler • Access to new data • General explanations (paradigms and other) – unsolved ‘big questions’ (Laudel and Gläser, 2012) Impacts of groundbreaking research on • • • • Own discipline or research field Several other research fields Opening up new research fields Merging fields/interdisciplinary areas of research Different perspectives • Selection of proposals - Forward look – – – – – Promise of opening up new avenues of research Enabling new research directions New perspectives Paradigm shifting, revolutionary Great uncertainty • Backward – Pin down what gave rise to the observed development – often a longer process and several contributors Difference between excellence and groundbreaking 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% LS Terttu Luukkonen: interviews with 24 ERC peer reviewers, Luukkonen, 2012 PE SH To a great or some extent Different definitions of groundbreaking and excellence ERC peer review panelists (Luukkonen, 2012) • Groundbreaking – Synonyms: pathbreaking, cutting-edge, frontier – Originality, novelty , revolutionary, paradigm shifting • Excellence – Originality, novelty, going beyond current state of the art, making a difference for the development of science, but also – Robustness of the research, methodological rigour, use of up-to-date methodology, coherent discussion of the research problem and purpose Development of ideas and scholars over time NR OF IDEAS & SCHOLARS SCHOLARS IDEAS Early adaptors Early Majorities Late majorities TIME Brown, 2012 Latecomers Project selection methods and cycles of ideas NR OF IDEAS & SCHOLARS National RCs National RCs Bibl. methods National RCs ERC HHMI Early adaptors Early Majorities Late majorities TIME Latecomers ERC peer review system Luukkonen, 2012 • Evaluation criteria • Quality of the peers • Panels have to consider feasibility and risks – Capabilities of the investigator – Instruments and equipment – Contingency plans – Avoid speculation and dilettantism – Put in context, tradition – Reasonable risk Laudel and Gläser on ERC, 2012 • “ERC grants have impact on research because, at least, some of them fund scientific innovations, the exploitation of recent discoveries, or answers to ‘big questions’ across all discipline groups” • The funded research has epistemic properties not usually met by grants from national funding agencies – – – – – – Contradicting the mainstream Addressing the community’s blindspots Linking otherwise separate communities Strategic & technical uncertainties Complexities in equipment, approaches Length of time it takes to conduct the research Project selection methods and cycles of ideas NR OF IDEAS & SCHOLARS National RCs National RCs Bibl. methods National RCs ERC HHMI Early adaptors Early Majorities Late majorities TIME Latecomers Bibliometric measures suggested Hörlesberger et al., 2013 • Novelty – Timeliness: how recent are the publications listed on the reference list of application – Similarity of proposal with emerging topics • Risk – Similarity of the proposed research to the investigator’s previous research – Interdisciplinarity • There has to be some body of publications in the field for the measures to be counted Project selection methods and cycles of ideas NR OF IDEAS & SCHOLARS National RCs National RCs Bibl. methods National RCs ERC HHMI Early adaptors Early Majorities Late majorities TIME Latecomers Varieties of peer review • Robustness – Use of international vs national experts – Independence of the panels – Their evaluation instructions and criteria • Further features: – – – – – Fine-grained vs. rough marking (Langfeldt, 2001) Remote reviews vs. or, in addition, panelists’ reviews Degree of interdisciplinarity of panels Panels rank or rate Evaluation criteria: ground-breaking vs. excellence Conclusions • Peer review conservative? • Not just one peer review, but many varieties in quality, criteria, organisation • The way peer review is organised and applied makes a difference • Peer review in combination with the conditions of the scheme can make a difference for progress of science • Quality of peer review provides legitimacy to the scheme and affects quality of the applicants • A risk that a thorough monitoring of peer review shifts the system towards more conventional proposals – short term indicators Thank you for your attention! more information: [email protected] Literature – – – – – – – – Braun, Dietmar. 2012. Why do scientists migrate? A diffusion model, Minerva, 50: 471-491. Grant, Jonathan and Allen, Liz. 1999. Evaluating high risk research: an assessment of the Wellcome Trust’s Sir Henry Wellcome Commemorative Awards for Innovative Research, Research Evaluation, 8: 201-204. Hörlesberger, Marianne, Roche, Ivana, Basagni, Dominique, Scherngell, Thomas, Francois, Claire, Cusax, Pascal, Schiebel, Edgar, Zitt, Michel, and Holste, Dirk. 2013. S concept for inferring ‘frontier research’ in grant proposals, Scientometrics, 97: 129-148. Langfeldt, Liv. 2001. The decision-making constraints and processes of grant peer review, and their effects on the review outcomes, Social Studies of Science, 31/6: 820-841. Laudel, Grit and Gläser, Jochen. 2012. The ERC’s impact on the grantees’ research and their careers (EURECIA Work package 4 summary report). January 2012. Luukkonen, Terttu. 2012. Conservatism and risk-taking in peer review: Emerging ERC practices, Research Evaluation, 21 (2012), No. 1, pp. 48–60. Nedeva, Maria. 2012. Peer review and path-breaking research: selection practices of research funding organisations. Unpublished. Wagner, Caroline S. and Alexander, Jeffrey. 2013. Evaluating transformative research programmes: A case study of NSF Grants for Exploratory Research Programme, Research Evaluation, 22: 187-197. UNI project: Universities, funding systems, and the renewal of the industrial knowledge base – a project funded by Tekes, 2012-2014; coordinated by Terttu Luukkonen; empirical data gathering with research group leaders in universities in Finland and the UK.