Les évaluations dans le premier appel d'offre du 6ème Programme-Cadre Thématique 7 : Citoyens et gouvernance dans une société de la connaissance Pascal Dissard Institut des Sciences de l'homme (ISH UMS 1798) / Pôle Europe Cellule Europe du Réseau des Maisons des Sciences de l'homme 14, avenue Berthelot. 69363. Lyon cedex 07. FRANCE Tel : +33 (0)4-72-72-79-36 / Fax : +33 (0)4-72-80-00-08 Pascal.Dissard@ish-lyon.cnrs.fr http://europe.ish-lyon.cnrs.fr/extranet/ P. Dissard - Evaluations PCRDT6 1
Les acteurs du processus d'évaluation -Evaluateurs (experts indépendants) - Evaluation scientifique des projets -Officiers scientifiques (fonctionnaires C.E.) - Contrôle de la procédure d'évaluation - Peuvent jouer le rôle de modérateur dans les réunions de consensus et de panel - Observateurs (experts indépendants) [optionnel] - Observation de la conduite et de l'exécution de la procédure d'évaluation P. Dissard - Evaluations PCRDT6 2
Processus d'évaluation Etape 1 Briefing des évaluateurs Etape 2 Evaluations individuelles Etape 3 Consensus Etape 4 Panel (optionnel) (auditions possibles) P. Dissard - Evaluations PCRDT6 3
Evaluation de projet : Notes (tous instruments) 5 ou 6 critères notés sur une échelle de 0 à 5 (possibilité de demipoints). Tous les critères ont le même poids (1) Notes 0 = Ne répond pas au critère / Manquant / Incomplet 1 = Faible 2 = Passable 3 = Bon 4 = Très bon 5 = Excellent P. Dissard - Evaluations PCRDT6 4
Evaluation d'un Réseau d'excellence (REX) (Critères) Critère 1 : "RELEVANCE" [Pertinence par rapport au Programme de travail] (Seuil individuel : 3/5) 1.1 The extent to which the proposed project addresses the objectives of the work programme Critère 2 : "POTENTIAL IMPACT" [Impact sociétal] (Seuil individuel : 3/5) 2.1 The extent to which Europe has a strategic need to strengthen S&T excellence on the topic by means of restructuring of the existing research capacities and the way research is carried out 2.2 The extent to which the goals of the network are, in that connection, suitably ambitious particularly, in terms of achieving European leadership and acting as a world force on this topic 2.3 The extent to which the proposal demonstrates a clear added value in carrying out the work at European level, does it take account of research activities at national level and under European initiatives (e.g.eureka) 2.4 The extent to which an effective plan for spreading excellence, exploiting results and disseminating knowledge, including to SMEs and to those outside the network 2.5 The extent to which the proposed approach is likely to have a durable structuring impact on European research P. Dissard - Evaluations PCRDT6 5
Evaluation d'un Réseau d'excellence (Critères) Critère 3 : "EXCELLENCE OF THE PARTICIPANTS" [Excellence des participants] (Seuil individuel : 3/5) 3.1 The extent to which the participants are currently conducting excellent research relevant to the topic of the network or are capable of important contributions to the joint programme of activities 3.2 The extent to which the participants are well suited to the tasks assigned to them 3.3 The extent to which the collectively have the necessary critical mass of expertise and resources to carry out the joint programme of activities successfully Critère 4 : "DEGREE OF INTEGRATION AND THE JPA" [Degré d'intégration du Réseau et programme commun d'activités] (Seuil individuel : 4/5) 4.1 The extent to which degree of integration justifies supporting the proposal as a network of excellence 4.2 The extent to which the joint programme of activities is sufficiently well designed to achieve the expected degree of integration 4.3 The extent to which the participating organisations have made a convincing commitment towards a deep and durable integration beyond the period of Community support P. Dissard - Evaluations PCRDT6 6
Evaluation d'un Réseau d'excellence (Critères) Critère 5 : "ORGANISATION AND MANAGEMENT" [Organisation et management du Réseau] (Seuil individuel : 3/5) 5.1 The extent to which the organisational structure of the network provides a secure framework for any necessary structural decisions to be taken 5.2 The extent to which the management of the network is demonstrably of high quality 5.3 The extent to which there is a well-considered plan for promoting gender equality in the network + SEUIL GENERAL = 20/25 P. Dissard - Evaluations PCRDT6 7
Evaluation d'un Projet Intégré (PI) (Critères) Critère 1 : "RELEVANCE" [Pertinence par rapport au Programme de travail] (Seuil individuel : 3/5) 1.1 The extent to which the proposed project addresses the objectives of the work programme Critère 2 : "POTENTIAL IMPACT" [Impact sociétal] (Seuil individuel : 3/5) 2.1 The extent to which the proposed project is suitably ambitious in terms of its strategic impact on reinforcing competitiveness (including SMEs) or on solving societal problems 2.2 The extent to which the innovation-related activities and exploitation and/or dissemination plans are adequate to ensure optimal use of the project results 2.3 The extent to which the proposal demonstrates a clear added value in carrying out the work at European level and takes account of research activities at national level and under European initiatives (e.g. Eureka) P. Dissard - Evaluations PCRDT6 8
Evaluation d'un Projet Intégré (Critères) Critère 3 : "S&T EXCELLENCE" [Excellence scientifique et technologique] (Seuil individuel : 4/5) 3.1 The extent to which the project has clearly defined objectives 3.2 The extent to which the objectives represent clear progress beyond the current state-of-the-art 3.3 The extent to which the proposed S&T approach is likely to enable the project to achieve its objectives in research and innovation Critère 4 : "QUALITY OF THE CONSORTIUM" [Qualité du consortium] (Seuil individuel : 3/5) 4.1 The extent to which the participants collectively constitute a consortium of high quality 4.2 The extent to which the participants are well-suited and committed to the tasks assigned to them 4.3 The extent to which there is good complementarity between participants 4.4 The extent to which the profiles of the participants, including those to be included later, have been clearly described 4.5 The extent to which the real involvement of SMEs has been adequately addressed P. Dissard - Evaluations PCRDT6 9
Evaluation d'un Projet Intégré (Critères) Critère 5 : "QUALITY OF THE MANAGEMENT" [Qualité du management] (Seuil individuel : 3/5) 5.1 The extent to which the organisational structure is well matched to the complexity of the project and to the degree of integration required 5.2 The extent to which the project management is demonstrably of high quality 5.3 The extent to which there is a satisfactory plan for the management of knowledge, of intellectual property and of other innovation-related activities Critère 6 : "MOBILISATION OF RESOURCES" [Mobilisation des ressources] (Seuil individuel : 3/5) 6.1 The extent to which the project mobilises the minimum critical mass of resources (personnel, equipment, finance) necessary for success 6.2 The extent to which the resources are convincingly integrated to form a coherent project 6.3 The extent to which the overall financial plan for the project is adequate + SEUIL GENERAL = 24/30 P. Dissard - Evaluations PCRDT6 10
Evaluation d'un Projet Spécifique de Recherche Ciblée (STREP) (Critères) Critère 1 : "RELEVANCE" [Pertinence par rapport au Programme de travail] (Seuil individuel : 3/5) 1.1 The extent to which the proposed project addresses the objectives of the work programme Critère 2 : " S&T EXCELLENCE" [Excellence scientifique et technologique] (Seuil individuel : 4/5) 2.1 The extent to which the project has clearly defined and well focused objectives 2.2 The extent to which the objectives represent clear progress beyond the current state-of-the-art 2.3 The extent to which the proposed S&T approach is likely to enable the project to achieve its objectives in research and innovation P. Dissard - Evaluations PCRDT6 11
Evaluation d'un Projet Spécifique de Recherche Ciblée (STREP) (Critères) Critère 3 : "POTENTIAL IMPACT" [Impact sociétal] (Seuil individuel : 3/5) 3.1 The extent to which the proposed project is likely to have an impact on reinforcing competitiveness or on solving societal problems 3.2 The extent to which the exploitation and/or dissemination plans are adequate to ensure optimal use of the project results 3.3 The extent to which the proposal demonstrates a clear added value in carrying out the work at European level and takes account of research activities at national level and under European initiatives (e.g. Eureka) Critère 4 : "QUALITY OF THE CONSORTIUM" [Qualité du consortium] (Seuil individuel : 3/5) 4.1 The extent to which the participants collectively constitute a consortium of high quality 4.2 The extent to which the participants are well-suited and committed to the tasks assigned to them 4.3 The extent to which there is good complementarity between participants 4.4 The extent to which the real involvement of SMEs has been adequately addressed P. Dissard - Evaluations PCRDT6 12
Evaluation d'un Projet Spécifique de Recherche Ciblée (STREP) (Critères) Critère 5 : "QUALITY OF THE MANAGEMENT" [Qualité du management] (Seuil individuel : 3/5) 5.1 The extent to which the project management is demonstrably of high quality 5.2 The extent to which there is a satisfactory plan for the management of knowledge, of intellectual property and of other innovation-related activities Critère 6 : "MOBILISATION OF RESOURCES" [Mobilisation des ressources] (Seuil individuel : 3/5) 6.1 The extent to which the project foresees the resources (personnel, equipment, finance ) necessary for success 6.2 The extent to which the resources are convincingly integrated to form a coherent project 6.3 The extent to which the overall financial plan for the project is adequate + SEUIL GENERAL = 21/30 P. Dissard - Evaluations PCRDT6 13
Evaluation d'une Action de Coordination (AC) (Critères) Critère 1 : "RELEVANCE" [Pertinence par rapport au Programme de travail] (Seuil individuel : 3/5) 1.1 The extent to which the proposed project addresses the objectives of the work programme Critère 2 : "QUALITY OF THE COORDINATION" [Qualité de la coordination] (Seuil individuel : 4/5) 2.1 The extent to which the research actions/programmes to be coordinated are of demonstrably high quality 2.2 The extent to which the coordination mechanisms proposed are sufficiently robust for ensuring the goals of the action P. Dissard - Evaluations PCRDT6 14
Evaluation d'une Action de Coordination (AC) (Critères) Critère 3 : "POTENTIAL IMPACT" [Impact sociétal] (Seuil individuel : 3/5) 3.1 The extent to which the proposal demonstrates a clear added value in carrying out the work at European level and takes account of research activities at national level and under European initiatives (e.g. Eureka) 3.2 The extent to which the Community support would have a real impact on the action and its scale, ambition and outcome 3.3 The extent to which the project mobilises a critical mass of resources in Europe Critère 4 : "QUALITY OF THE CONSORTIUM" [Qualité du consortium] (Seuil individuel : 3/5) 4.1 The extent to which the participants collectively constitute a consortium of high quality 4.2 The extent to which the participants are well-suited to the tasks assigned to them 4.3 The extent to which the project combines the complementary expertise of the participants to generate added value with respect to the individual participants' programmes P. Dissard - Evaluations PCRDT6 15
Evaluation d'une Action de Coordination (AC) (Critères) Critère 5 : "QUALITY OF THE MANAGEMENT" [Qualité du management] (Seuil individuel : 3/5) 5.1 The extent to which the project management is demonstrably of high quality 5.2 The extent to which there is a satisfactory plan for the management of knowledge, of intellectual property and of other innovation-related activities Critère 6 : "MOBILISATION OF RESOURCES" [Mobilisation des ressources] (Seuil individuel : 3/5) 6.1 The extent to which the project provides for the resources (personnel, equipment, finance ) necessary for success 6.2 The extent to which the resources are convincingly integrated to form a coherent project 6.3 The extent to which the overall financial plan for the project is adequate + SEUIL GENERAL = 21/30 P. Dissard - Evaluations PCRDT6 16
Questions transversales (Evaluations tous instruments) Questions transversales à prendre en compte (non notées): - Questions de "gender" associées au projet? - Identification des questions liées à l'éthique et à la sécurité? - Plan de dissémination des résultats? - Identification des liens avec les questions d'éducation? - Si Pays Tiers impliqués dans le projet ===> justifier leur participation et veiller à leur bonne intégration dans les activités P. Dissard - Evaluations PCRDT6 17
Note > Seuil général Note < Seuil général = Seuils individuels 5 Evaluations Réseaux d'excellence * 16 REX (11 coord. françaises, 5 coord. étrangères) * 5 projets > au seuil général, 11 rejetés Seuil 4 général Notes 3 2 1 0 Relevance Impact Excellence Integration Management 1 - Pour l'ensemble des projets, faiblesse relative du critère "Impact potentiel" par rapport à "Excellence des participants" et à "Organisation et management" 2 - Différence significative sur tous les critères entre les projets au-dessus du seuil général (shortlist) et les projets en-dessous (projets rejetés) Critères d'évaluation P. Dissard - Evaluations PCRDT6 18
Note > Seuil général Note < Seuil général = Seuils individuels 5 Evaluations Projets Intégrés * 12 PI (4 coord. françaises, 8 coord. étrangères) * 4 projets > au seuil général, 8 rejetés Seuil général 4 3 1 - Pour l'ensemble des projets, aucune différence significative entre les critères d'évaluation Notes 2 1 0 Relevance Impact S&T Excellence Consortium Management Resources 2 - Différence significative sur 5 critères entre les projets au-dessus du seuil général (shortlist) et les projets endessous (projets rejetés). Pas de différence relevée pour le critère "Management" Critères d'évaluation P. Dissard - Evaluations PCRDT6 19
6 Résultats individuels (NoE / Critères) Quality of management Quality of consortium S&T Excellence Potential impact Relevance Réseaux d'excellence 5 4 Notes 3 2 1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Projets rejetés Shortlist P. Dissard - Evaluations PCRDT6 20
Résultats individuels (PI / Critères) 6 5 Mobilisation of resources Quality of management Quality of consortium Potential impact S&T Excellence Relevance Projets Intégrés 4 Notes 3 2 1 0 1 2 3 4 5 6 7 8 9 10 11 12 Projets rejetés Shortlist P. Dissard - Evaluations PCRDT6 21
Principaux points problématiques (REX) 1 - RELEVANCE - Couverture des objectifs du programme de travail 2 - POTENTIAL IMPACT - Plan de dissémination des résultats et liens avec la société civile (impact limité à la communauté scientifique) 3 - EXCELLENCE OF THE PARTICIPANTS - Monodisciplinarité - Justification des participations 4 - DEGREE OF INTEGRATION AND THE JPA - Intégration des activités (activités "éclatées") et des partenaires 5 - ORGANISATION AND MANAGEMENT - Description claire de la structure de management P. Dissard - Evaluations PCRDT6 22
Principaux points problématiques (PI) 1 - RELEVANCE - Couverture des objectifs du programme de travail - Objectifs trop ciblés 2 - POTENTIAL IMPACT - Plan de dissémination des résultats et liens avec la société civile (impact limité à la communauté scientifique) 3 - S&T EXCELLENCE - Monodisciplinarité - Justification des participations et liens entre les activités 4 - QUALITY OF THE CONSORTIUM - Monodisciplinarité - Gender 5 - QUALITY OF THE MANAGEMENT - Description claire de la structure de management 6 - MOBILISATION OF RESOURCES - Justification de la demande budgétaire - Budgets sur-évalués P. Dissard - Evaluations PCRDT6 23