Final Evaluation Inception Report Regional Sustainable Management of Endemic Ruminant Livestock in West Africa (PROGEBE Regional Coordination Unit) GEF/UNDP/AfDB ATLAS Award Id: Project Id: PIMS: GEF SEC Project ID: September 2014 Stephanie Hodge, Canada, GEF component Roel Bosma, Netherlands, AfDB Component
ACRONYMS AND ABBREVIATIONS APR Annual Project Review AWP Annual Work Plan BD Biodiversity CBO Community Based Organization CC Climate Change CP Country Program CPAP Country Program Action Plan CTA Chief Technical Advisor DG Director General DOE Department of Environment EB Executive Board FE Final Evaluation GEF Global Environment Facility HR Human Resources INRM Integrated Natural Resources Management IW International Waters LD Land Degradation MDGs Millennium Development Goals M&E Monitoring and Evaluation MTR Mid Term Review NCCD National Committee to Combat Desertification NAPCD National Action Program to Combat Desertification NBSAP National Biodiversity Strategy and Action Plan NCSD National Committee on Sustainable Development NEX National Execution Modality NGO Non Governmental Organization NPD National Project Director NPM National Project Manager NPMC National Project Management Committee PB Project Board PES Payment for Environmental Services PIR Project Implementation Review PMC Planning and Monitoring Committee 4 RTU Project Management Unit POPP Program and Operations Policies and Procedure PPC Provincial Planning Council PPM Provincial Project Manager PPMC Provincial Project Management Committee PRTU Provincial Project Management Unit PPR Project Progress Report PPRC Provincial Project Review Committee PRA Participatory Rural Appraisal PRF Project Result Framework ProDoc Project Document PTD Participatory Technology Development QPR Quarterly Project Report TC Technical Committee ToR Terms of Reference UNCCD United Nations Convention to Combat Desertification UNCED United Nations Conference on Environment and Development UNDAF United Nations Development Assistance Framework UNDP United Nations Development Program UNESCO United Nations Educational, Scientific and Cultural Organization UNFCCC United Nations Framework Convention on Climate Change II
PROJECT SUMMARY TABLE Project Title: Regional Project on Sustainable Management of Endemic Ruminant Livestock in West Africa (PROGEBE) GEF Project ID: at endorsement (Million US$) PIMS 1119 AfDB Project SAP Code AfDB Grant number AfDB Loan number P Z1 AAE 004 (ITC, The Gambia et Guinea) P Z1 AAE O10 (Senegal) P Z1 AAE O09 (Mali) 2100155006866 (CIT, The Gambia & Guinea) 2100150016493 (Mali) 2100150016494 (Senegal) AfBD total: broken down as: $29.767 239M (i) AfDB Grant $15M (10,240,000 UA) (ii) AfDB loan $14.3 (9,760,000 UA) UNDP Project ID: 00046252 GEF financing: $10.0M UNOPS Project ID 57220 UNDP Mali Project Award 54953 Country: Gambia, Guinea, Mali and Senegal Region: West Africa IA/EA own: 0 Government: Focal Area: Biodiversity Other ITC, ILRI AfDB Division FA Objectives, (OP/SP): Executing Agency: Other Partners involved: OSAN4/OSAN2 GEF Strategic objectives: BD 2 Mainstreaming Biodiversity in production. UNOPS/ITC ILRI, ITC, FAO $3.4M Total co financing: $13.4 Total co financing: Total Project Cost: $32.6 $42.6M ProDoc Signature (date project began): 29 June 2007 AfDB Grant signature date AfDB Loan signature date 30 June 2006 16 October 2006 (Operation al) Closing Proposed: GEF: 2016 Actual: GEF: 2015 Date: AfDB: 2013 Grant: June 2014 Senegal Loan: October14 Mali Loan: December14 III
TABLE OF CONTENTS (PROGEBE Regional Coordination Unit)... I September 2014... I 1. PROJECT BACKGROUND... 2 2. RATIONAL - UNDP /GEF EVALUATION POLICY /EXPECTATION OUTCOMES... 5 3. METHODS... 6 4. OUTPUTS OF EVALUATION... 98 5. SUPPORT REQUIRED FROM THE RTU... 9 6. KEY CHALLENGES EXPECTED AND HOW TO TACKLE THEM... 9 ANNEX 1 EVALUATION MATRIX RELEVANCE, EFFICIENCY AND EFFECTIVENESS... 0 ANNEX 2 SAMPLE SURVEY QUESTIONAAIRE FOR PROJECT FORMULATION /DESIGN AND IMPLEMENTATION ASPECTS... 02 ANNEX 3 - REPORT - KEY FINANCIAL TABLES FOR GEF FINAL REPORTING.. 2225 ANNEX 4 GEF REQUIRED PROJECT RATINGS - ASSESSMENT AS PER UNDP /GEF EVALUATION GRID... 2326 ANNEX 5 AGENDA Programme - Final Evaluation PROGEBE (14 SEPTEMBER - 3 OCTOBER, 2014)... 0 INCEPTION REPORT / DRAFT Page 1
1. PROJECT BACKGROUND Project context In partnership with the Global Environment Facility (GEF), the African Development Bank (AfDB) and United Nations Development Programme (UNDP), the Governments of The Gambia, Guinea, Mali and Senegal, are currently implementing an integrated program which aims to remove existing barriers to the in-situ conservation of three priority endemic ruminant livestock species N dama cattle, Djallonke sheep, and the West African Dwarf goat and improve their productivity. In addition, the project will develop and implement models for community-based conservation and management of critical habitat for these species, thereby demonstrating strategies for preserving the unique genetic trait/habitat complexes that are of global significance. The project design is experimental, developing and testing an integrated approach to livestock conservation and management that simultaneously addresses livestock breeding and productivity, market development and economic policies, incentives and distortions, traditional and evolving patterns of resource use and land tenure, policies and legal frameworks, and information sharing and communication at the national, regional and international levels. The largest remaining populations of endemic livestock in the sub-region of Senegal, Gambia, Mali and Guinea consist of N dama cattle, Djallonke sheep and West African Dwarf goats. Although numbers of these breeds are still relatively high, their future is in jeopardy due to varied and complex threats, which can be broadly grouped into three primary categories: 1) destruction and degradation of habitat critical for endemic ruminant livestock; 2) cross-breeding between endemic ruminant livestock and exotic livestock breeds; and finally, 3) abandonment of endemic ruminant livestock raising due to production and market constraints. In addition, very little information exists on actual populations or rates of crossbreeding of these targeted endemic ruminant livestock breeds, so that the exact magnitude of the threat is unclear. Project Areas The project target zone consists of eastern Gambia, southern and southeastern Senegal, western and southern Mali, and central and southern Guinea. This transboundary zone consists of four vegetative formations, dominated by wooded savannas, as well as shrub savanna, open forest, and riparian gallery forests. The tree strata is dominated by species such as Daniella oliveri, Anogeissus leocarpus, Khaya senegalensis, Burkea africana, Bombax costatum, Pterocarpus erinaceus, Terminalia macroptera, Combretum glutinosum, Enteda africana, Isoberlina doka, Detarium senegalensis, etc. Although the vegetative formations are fairly similar across this transboundary zone, its topography is more varied. In Guinea, the landscape is highly variable and consists of rolling plains and plateaus broken up by the Fouta Djallon and Nimba Mountains. Southeastern Senegal is dominated by a high plateau and frequent hills, while in Gambia and Mali, the landscape is more flat. Within these landscapes, the project has selected twelve primary pilot sites in which to implement field level interventions, as well as eight secondary sites for replication of selected activities. These sites represent a wide range of natural ecological conditions, modes of resource management (including sedentary agro pastoral systems and migratory grazing systems - transhumance), and degree of prior human induced impact and current threats to ecosystems. Project Funding Two major donors (The AfDB and the GEF, with UNDP as the GEF Implementing Agency) have selected different executing agencies, but common implementation arrangements. The AfDB component is executed at regional level by the International Trypanotolerance Centre (ITC) based in the Gambia and at national level by the national governmental agencies responsible for livestock in the four project countries Department of Livestock Services (Gambia), Direction Nationale de l Elevage (Guinea), Direction National de l Elevage et de la Pêche (Mali), Direction de l Elevage (Senegal). The GEF component is executed by the United Nations Office for Projects Services (UNOPS), which has overall responsibility for the realization of the project s GEF outcomes and is responsible for facilitation of operational procedures with UNDP, the national governmental agencies responsible for livestock in the four project countries Department of Livestock Services (Gambia), Direction Nationale de l Elevage (Guinea), Direction National de l Elevage et de la Pêche (Mali), Direction de l Elevage (Senegal) and co-financing partners. ITC and UNOPS play a leading the role in the coordination of INCEPTION REPORT / DRAFT Page 2
project implementation, together with the project Regional Coordination Unit (RCU) based at ITC in the Gambia. Objectives and Scope The GEF and AfDB have their own project documents: the PRODOC for UNDP/GEF and the Project Appraisal Report for the AfDB. AfDB developed its 2005 project Appraisal Report on the basis of the GEF Approved Brief and Executing Summary from July 2004. Both documents departed from the same official documents that were produced to obtain the GEF Council approval of the project. They formulated a different main objective, different specific objectives and formulated the outcomes or outputs differently (Table 1). These difference result from different schools of thought on development cooperation/assistance, such as Research Impact Pathway and Outcome Mapping; the latter was probably used by UNDP but not by AfDB. AfDB s output 1 englobes UNDP/GEF s outcome 1 and 2, and AfDB s output 2 englobes UNDP/GEF s outcome 3 and 4, while output 3 and outcome 5 overlap (project management). Component B of the last Output is in fact a supportive activity, and not an outcome, and will be evaluated through its effectiveness, efficiency, coherence and beneficiary s participation, and not through its outcome. The evaluators will assess the results of component A of Outcome 5 (Output 3): sub-regional system for cooperation, information exchange, and coordinated support for the sustainable management of endemic ruminant livestock is established and operational. The final goal of both donors partly overlaps: improved rural economy versus poverty reduction and improved food security, but UNDP adds the biodiversity aspect. It goes without saying however, that there might be improvements of the underlying causes of poverty and food insecurity through biodiversity management and in the local environment in which people s lives depend on. An integrated logical framework was developed in consultation with key partners to facilitate and harmonize the implementation of the activities and communication with stakeholders. The integrated logical framework was approved by the Mid-Term Evaluation. Table 1: ProGeBe s objectives and outcomes as defined by AfDb and UNDP UNDP AfDB Main objective Specific objective Main Outcomes / Outputs Ensure sustainable populations of targeted endemic ruminant livestock breeds in four West African countries in order to improve rural economies and ensure the conservation of these breeds and their globally unique genetic traits. Establish effective models for community based management of endemic ruminant livestock and their habitat at project pilot sites, and strengthen production, market, and policy environments in support of these breeds. 1. Production and productivity of endemic ruminant livestock is sustainably improved; 2. Commercialization and marketing systems of endemic ruminant livestock and livestock products are strengthened ; 3. Natural resources in project pilot sites conserved and sustainably managed for the benefit of endemic ruminant livestock, ecosystem services, and human livelihoods ; 4. Legal, policy and institutional frameworks established at the local, national, and sub-regional level for in-situ conservation of endemic ruminant livestock ; 5. Project management. Contribute to poverty reduction and improved food security in West Africa. Conserve the biodiversity of endemic ruminant livestock and increase their productivity. 1.Improvement of endemic livestock production systems ; 2. Conservation of the natural habitat of endemic livestock ; 3. Project management. Project Log Frame The vetted project log frame (MTE) will be the starting point for this evaluation (Annex 6). The project log frame has been reconstituted and reviewed during the MTE. It outlines action and criteria around six strategic interventions lines (Outcomes) identified in the integrated logical framework are as follow: 1- SIL1: Preservation of genetic characterization and improvement of production and productivity of ERL 2- SIL 2: Improvement of the valorization of the ERL and its products 3- SIL 3I: Sustainable management of ERL and its ecosystem INCEPTION REPORT / DRAFT Page 3
4- SIL 4: Legal, policy and institutional frameworks 5- SIL 5: Cooperation, knowledge management, exchanges and coordination 6- SIL6: Project management. Project Duration The project was supposed to be implemented over ten-year period for the GEF component and 6 years period for the AfDB component. The PRODOC was signed in 29 June 2007 and the AfDB Grant and Loan agreements were respectively signed in 30 June 2006 and 16 October 2006. However, implementation of the project only started effectively in January 2008. The inception workshop was held in January 2009. The expected closing dates were end of 2016 for the GEF and end of 2013 for the AfDB. These closing dates have been revised to 30 th June 2014 for AfDB Grant (RCU, Gambia, Guinea, 31 st October 2014 for Senegal AfDB Loan, 31 st December 2014 for Mali AfDB Loan and early 2015 for GEF component. Project Management The regional coordination unit for the project (RCU) is based in the Gambia hosted by ITC. The RCU works in close collaboration with ITC, UNOPS and the government s agencies to ensure effective implementation of the project. At national level, the project is managed through four national coordination units (UNC) and 12 site teams. As for the project s co-financiers and partners, ITC, FAO, CIRDES, ILRI, they are also expected to provide advice on research, training needs and management of the project. The International Livestock Research Institute (ILRI) had been selected in the PRODOC to be the main contractor for the research field work implementation. ILRI s HQ is in Kenya, with the coordinator for ILRI s project activities based in Mali. The collaboration between ILRI-UNDP-UNOPS is covered by a MoU. The International Center for Livestock Research and Development in Subhumid Zones (CIRDES) had been identified in the AfDB appraisal report to be the partner for cryo-conservation activities. The partnership with CIRDES has not been established although several attempts were made during the project period. In collaboration with FAO, the project has led the establishment of a sub-regional focal point for the management of AnGR in West and Central Africa. An electronic discussions platform DAD-NET-WCA moderated by the project has been established within this framework In addition, a study on transhumance impact on the management of AnGR has been funded by FAO and its implementation being prepared under the leadership of ITC. At national level, several partnership protocols have been signed with national institutions for the implementation of activities such as technical and institutional capacity building, environmental monitoring, land use plans development, etc. Regional Steering Committee The project s first Regional Steering Committee meeting took place in February 2008 and the last one in late February 2014. Project Mid-Term Evaluation The MTE of the project took place in two steps with an overall evaluation in mid-2011 and a complementary assessment in late 2011/early 2012. The MTE endorsed the consolidated log frame and rated the overall project implementation successfully. The final review will take note of the MTE key points, conclusions and lessons learned, including: 1. The formulation of the project has followed a long process, but the project is still relevant and responds to the needs and priorities of beneficiary communities and the AfDB and GEF objectives. 2. Implementation and performance are mainly satisfactory. 3. There is a good level of stakeholder participation and beneficiaries ownership of the project, but a more precise analysis is required. INCEPTION REPORT / DRAFT Page 4
4. The efficiency of administrative, financial and accounting procedures has improved. Discrepancies in regional staff remuneration should be addressed. 5. The planning of ILRI activities lack tangible deliverables, although their contribution clearly adds value. 6. The monitoring and evaluation system defined at the inception of the project is functional. The computerized M&E system is not yet fully operational and monitoring the achievement of results is still quite limited. 7. The sustainability of the capacity built by the project needs to be enhanced. 8. The quality of human resources recruited to start the project is key to achieve project objectives. 9. There is need to fully take into account the changing socio-cultural intervention context and implementation modalities in order to set realistic objectives. 10. The project implementation approach through the faire faire requires a functional mechanism to monitor partner interventions so as to introduce timely corrective measures; it also takes some patience to negotiate and start the protocols. 11. A strong involvement of line Technical Departments in the supervision and technical monitoring of NCU activities will contribute to improving project performance and is also a token for sustaining achievements. 12. As concerns procurement of goods and services, the combined use of Bank procedures and those of national entities requires strengthening the quality of monitoring files to avoid any counter-performance. The achievements since the MTE on these 12 points will be documented by the FTE. 2. RATIONAL - UNDP /GEF EVALUATION POLICY /EXPECTATION OUTCOMES In line with UNDP / GEF Evaluation Policy, GEF funded projects are monitored and evaluated regularly. Through a TE exercise, GEF aims to promote accountability for achievement of GEF objectives through the assessment of results, effectiveness, processes and performance of the partners involved in the GEF supported activities to date and by making recommendations on improving likelihood for expected outcomes by making adjustments if needed. The results need to be monitored and evaluated for their contribution to global environmental benefits. The TE is intended to assess the relevance, performance and success of the project and support changes needed at mid term. In general, the Final Evaluation (FTE) will explore the following criteria: Relevance: the extent to which the planned activities were suited to local and national development priorities and organizational policies, including changes over time; Effectiveness: the extent to which an objective has been achieved or how likely it is to be achieved; Efficiency: the extent to which results have been delivered with the least costly resources possible, also called cost effectiveness or efficacy; Results: the positive/negative and foreseen/unforeseen changes to and effects produced by a development intervention to date. (In GEF terms, results include direct project outputs, short-tomedium-term outcomes and longer term impact, including global environmental benefits, replication effects and other local effects); Sustainability: the likely ability of an intervention to continue to deliver benefits for an extended period of time after completion. Projects need to be environmentally, as well as financially and socially, sustainable. The core of the proposed approach is the use of an Evaluation Matrix (Annex 1) which will: Ensure a clear and common understanding by researchers, clients and stakeholders of the approach and methods to be used in the evaluation. Ensure consistency in the approaches used for the evaluation themes and comparability of the findings. Where possible, inter-connected questions/issues will be cross-referenced in the matrix. Set out clearly the sources of information and chain of reasoning used in deriving the evaluation conclusions, facilitating evidence-based findings and recommendations. INCEPTION REPORT / DRAFT Page 5
The evaluation matrix and its related products the questionnaires and indicators will also provide a framework for the final report By the end of the field data collection mission, the extended evaluation team (National partners, International consultant, National consultants, UNDP, GEF/RTA and RTU and project local beneficiaries) will have answered core questions: what are we doing (design), what has changed, are we doing the right thing, and what can we do to make it better (lessons learned ). The draft programme of the consultation (including country based agenda) is provided in Annex 5 (Sept October 2014). An inception dialogue with the Evaluation team will be held shortly after the international consultants have arrived. 3. METHODS The evaluation methods are developed in accordance with the official 2012 GEF guidelines for conducting evaluations (GEF IEO website). In addition, the methods are closely guided by the specific criteria outlined in the Terms of Reference (TOR). i The TE evaluation is expected to document the project and generate evidence-based information for the implementing partners that is credible, reliable and useful based on a mixture of participatory methodologies. As per the TOR, the overall objective of the evaluation is to review project practice (management, implementation, monitoring). The evaluation entails a complex process involving research with institutional and non-institutional actors and working at different levels across the project and administration. Specifically, the evaluation will be frame using the following criteria: project relevance, efficiency and effectiveness, coherence, beneficiary s participation /sustainability and impact ii. In the evaluation matrix, a set of questions, verifiable indicators, data sources and data collection methods covering each of these criteria have been drafted in the evaluation matrix (Annex 1A). During the initial workshop with the RCU, in case some of the criteria of the LogFrame can t be verified, it might be useful to review the criteria given in the LogFrame and the initial criteria given by UNDP and AfDB (Annex 1B). The combined evaluation phases (desk study, field trips, consultation exercises including focus group work with key stakeholders and final beneficiaries) will provide the evaluation team with data, resources and insights to undertake a final analysis to trends, changes, uses and demand for instilling integrated natural resources management related products and ecosystem based services in the region. Methods to be used are specified hereafter. The evaluation will be carried out in three stages: 1. Documentation review (desk study): The list of project documentation is to be reviewed. These documents will be availed by Project Office and/or GEF/UNDP, including the Country Office and the Regional Coordination Unit. The project will be requested to provide the key financial tables (Annex 3), This will allow the evaluation team to assess the efficiencies as required by the TOR. 2. Data collection in the project region: Adaptation of predesigned survey tool through face to face meetings: A set of question have been developed to specifically interview local stakeholders. The survey tool will be technically deployed by the evaluators in consultation with the local stakeholders. The questions will be developed based on the guiding template for asking questions that can be adapted for questioning different stakeholders in different context. Interviews: The evaluator will consult and gather information locally through meetings with the various stakeholders (Government partners, NCUs, RTU, GEF/UNDP, AfDB, NGOs and private sector representatives). Guidelines for the interviews will be designed (see Annex 2 for the drafts). To assess, at the level of the final beneficiaries the questions, what has changed and what can we do to make it better (lessons learned) the team will request them to formulate stories/statements of Most Significant Change. The other local beneficiaries will rank these and formulate (add) statements on Most Significant Change on what are we doing (design), what has changed, INCEPTION REPORT / DRAFT Page 6
are we doing the right thing, and what can we do to make it better (lessons learned ). The collected statements / stories will be ranked by the national and regional project teams. Stakeholder workshops / focus group discussions: are important to evaluate for capacity strengthening with the beneficiaries directly including in the institutions strengthened or set up, as well as to assess results and impact. Focus groups are very important for this evaluation due to its large scope. The evaluators propose to undertake workshops with the RTU and the NC teams in particular. Other focus groups will be convened as per the mission program outlined in annex 5. Field visits will be made to the pilot sites or proposed demonstration sites to meet the final beneficiaries. These visits and meeting are important to verify claimed changes, results and impacts (See proposed program Annex 5). 3. Analysis of the information collected: The third evaluation step is an objective and evidence based assessment of project performance. This will be carried out, based on expectations set out in the approved Project Logical Framework/Results Framework (Inserted as PDF attachment). This document is the baseline and overall assessment framework to be considered as it provides performance and impact indicators for project implementation along with their corresponding means of verification. Ratings will be provided according to the GEF /UNDP obligatory rating scales (Annex 4) iii. 4. Drafting of the final Terminal Evaluation Reports: The fourth step will be the consolidation of the analysis into two final reports which will be shared with all key project stakeholders for comment and revisions. The consultants will accept leadership on one report each however the analysis will be done as a team and each consultant will read the others report as an independent reviewer and collaborator to ensure consistency in findings. This process is iterative and will require intense and close work between evaluator and evaluation stakeholders. Thus a mixture of quantitative and qualitative methodologies will be employed, including: desk study/literature review, face to face consultations and focus groups with stakeholders (government, beneficiaries and boundary partners, GEF, UNDP, RTU, and National partners). It will also rely heavily on field visits to project sites for observation and triangulation in consultation with local beneficiaries. EVALUTION TEAM The external evaluation team is comprised of two independent and international experts (s). Ms. Hodge, hired by UNOPS will focus on the GEF reporting requirements and overall capacity strengthening and institutional aspects, while Mr. Bosma will mainly address the ELR related issues. Together they will address the cross-sectional issues such as sustainability and gender. They will use the respective sections in their reports to UNOPS and AfDB respectively. The project management unit, the RTU, UNDP and the GEF RTA will provide support. The team will work closely with the extended project team (UNDP, GEF, national stakeholders) and with beneficiaries (if possible or feasible-institutional groups) to plan the evaluation and to review the recommendations for implementation. The owners of the evaluation must take forward the recommendations; therefore, their involvement was essential to forming conclusions and making implementable recommendations. To capitalize on the comparative advantage of the skill of the evaluators the international expert hired by UNOPS will lead the drafting of the final reports for UNOPS, while the international expert hired by AfDB will lead the drafting of the final reports according to the outline given by AfDB. The international experts assigned each other focal areas for some aspects of the FTE. For example, Ms Hodge, hired by UNOPS will focus on the institutional aspects, while Mr. Bosma will mainly address the ELR related issues. Together they will address the cross-cutting issues such as sustainability and gender, as these relate to both technical and institutional aspects. The experts will share responsibilities for: Focus groups, individual interviews and other techniques for information gathering. Observations during the visits of the project sites and meetings with the local beneficiaries. This will help evaluate the demonstrated results and physical progress of projects as evidence. INCEPTION REPORT / DRAFT Page 7
DESK STUDY A desk study, including a comprehensive review of the project documentation provided by the RTU and the local UNDP office, is conducted in advance and ongoing during the actual field visit. This study supports the development of the hypothesis on the situation and of concise methods and division of responsibilities for data collection. These documents help also to understand the situation of the ELR and to describe how the project intervention was designed and is performing to date. The core documentation iv reviewed included UNDP and GEF TE evaluation policy, the project document, annual project reports, project steering committee minutes and decisions, budgets, work plans, files, reports, Project implementation reports (PIRs), UNDP guidance documents, national legislation relevant to the project and any other material considered useful by members of the evaluation team. - STAKEHOLDERS CONSULTATIONS The stakeholders to be consulted include: UNDP Program Manager, members of the Project Board and Steering Committee (SC), Project Management Unit, Representative from the Ministries of Agriculture, other relevant ministries, National NGO representation and the primary project beneficiaries in the project area (RTU should prepare a list in advance of international mission). Semi-structured interviews will be designed to ensure that all aspects of project are covered. Focus group discussions, using guiding questions with project beneficiaries and a questionnaire, were developed for this purpose. Participatory evaluation techniques and other approaches for the gathering and analysis of data are central. The following stakeholders will be consulted (see program next section) during the mid-term evaluation: - Project leaders: UNDP Program Manager, members of the Project Board and Steering Committee (SC), Project Management Unit, Representative from the Ministries of Agriculture, other relevant ministries. - Beneficiaries (key-stakeholders): breeders associations, management committees of infrastructures, women groups, management committees of community land use plans or POAS, owners of N Dama cattle and Djallonké sheep and goats; livestock traders and butchers; and communities or individuals having benefited from abattoir, dairy processing unit or market. - Boundary partners: for example private veterinarians, local government administration and NGOs, and those who personally have not benefitted directly from an activity of the project but were interested or impacted through e.g. the POAS. - Other stakeholders: ILRI, CIRDES, FAO, UNDP-GEF and others to be identified together with RCU and NCUs. Table 2: The deliverables of the evaluation team Deliverable Content Timing Responsibilities Inception Report Evaluation team provides clarifications on timing and No later than 2 weeks before the evaluation mission. Evaluators submit to RCU CC to UNOPS, ITC & AfDB method, presents the agreed mission plan Mission debriefing Initial Findings End of evaluation mission To National and Regional Coordination Units Draft Final Reports Full reports (per AfDB and UNDP-GEF templates) Within 2 weeks of the end of the field mission Sent to RCU, NCU, UNOPS, ITC, AfDB, UNDP RTA, GEF OFPs Final Reports* Revised reports Within 1 week of receiving Sent to RCU for submission to AfDB Presentation Results of the Final Evaluation comments on draft Date to be agreed upon by the interested parties. and uploading to UNDP ERC. In the Gambia. to be organized by RCU INCEPTION REPORT / DRAFT Page 8
4. OUTPUTS OF EVALUATION A PowerPoint presentation will be given in English before the international consultants depart the project region (See Table 2 below). The draft of the Final Evaluation Report (FTR) in English will be submitted within two weeks after the end of the field mission; see FTE agenda Annex 5). The final report will be submitted within a week of receiving written comments on the drafts from AfDB/UNDP/GEF and partners. If there are any significant discrepancies between the impressions and findings of the evaluation team and stakeholders, these will be explained and attached to the final report. The main document of the Final Evaluation Report will have approximately 40 pages, excluding annexes, according to the attached detailed breakdown given in the TOR and the two Technical proposals). The main reports will be in English and in French will be prepared and submitted in MS Word, with tables in Excel where necessary. Annexes may be either in French or English. 5. SUPPORT REQUIRED FROM THE RTU The RTU shall provide backstopping and logistical support to the evaluation team during design (inception report writing) and implementation. This includes arranging the program logistics and providing an office to enable the team to conduct analysis and reviews. The RTU will support data collection as instructed to facilitate a global standard for GEF evaluation including the compiling of financial data. The RTU are evaluation key informants and will complete a management questionnaire. They will engage in a RTU evaluation self assessment workshop. They will provide data on the financial siltation and compile information on the status of activities. 6. KEY CHALLENGES EXPECTED AND HOW TO TACKLE THEM Key challenges anticipated are logical constraints to meet all project stakeholders, to visit all project pilot sites considering the rainy season as well as the on-going Ebola-epidemic, and the perceived language barriers. The first will be tackled by splitting the group into teams to cover all stakeholders, and the second by selecting accessible sites. Guinea will not be visited; the NTU will come to Bamako to meet the evaluators; other involved stakeholders can be interviewed by phone if needed. Language barriers will be overcome by involvement of the local project staff, and preferably by French/English speaking beneficiaries. The TOR defined two points which the evaluators can t fulfil or can fulfill partly: The evaluators cannot be held responsible for the requirement to attach the Updated Implementation Progress and Results Report (IPR) of which the date should be the same as the PCR mission (see Annex F of the Standard Request ). The IPR falls outside their competence and responsibility. The ToR requests evidence-based credible and reliable information. Upon request the Project Team specifies: Reliability depends on acceptable scientific criteria and relevant standards for similar work. Please be aware that a 20-day evaluation of which at least one-third (7) is spend travelling to visit 11 sites and at least 4 towns/locations to meet project staff or leaders, at several hours to a day traveling from each other, cannot reply to scientific standards. The latter should come from the project monitoring or from the scientific studies from ILRI. At the most the evaluators can verify the information provided in documents through interviews at the 15 locations with perhaps 60 different individuals/groups on a variety of subjects. Not each respondent can be familiar with each subject, and thus, the sample size will be insufficient for a scientific standard under expected high variability at 11 sites and 4 countries. The latter would require a sample size far more than 100 and for this a separate data collection is needed; this can be organized using reliable local interviewers, but 20 days is largely insufficient to do the analysis of such a dataset, next to the required meetings with relevant stakeholders and the writing of reports in both French and English. INCEPTION REPORT / DRAFT Page 9
ANNEX 1A EVALUATION MATRIX RELEVANCE, EFFICIENCY AND EFFECTIVENESS Evaluative Criteria Questions Indicators Sources Methodology Relevance: How does the objectives of the project relate to the main objective of the AfDB sector and GEF focal area, and to the environment and development priorities at the local, regional and national levels? Is the Project relevant to the multilatéral environmental conventions MEA international Conventions objectives? Is the project Relevant the GEF four focal Areas? LD SP1; IW SP3; CC SP6 BD SP4 Is the project relevant to member countries environment and sustainable development objectives? Is the project addressing the needs of target beneficiaries at the local and national and regional levels? How does the project support the objectives of MEA conventions? How does the project support the GEF focal area and strategic priorities How does the project support the environment and sustainable development objectives of the Country? Is the project Country driven? What was the level of stakeholder participation in project design? What was the level of stakeholder ownership in Implementation? Does the Project adequately take into account the national realities, both in terms of institutional and policy How does the project support the needs of relevant stakeholders? Has the Implementation of the project been inclusive of all relevant Stakeholders? Were local beneficiaries and stakeholders adequately involved in project design and In line with MEA Conventions priorities and whether areas of work are incorporated in project design Level of implementation of the three MEA conventions in country and contribution of the project priorities and areas of work of other conventions incorporated in project design Extent to which the project is actually implemented in line with incremental cost argument Existence of a clear relationship between the project objectives and GEF focal area LD SP1; IW SP3; CC SP6 BD SP4 s Degree to which the project supports National environmental objectives Degree of coherence between the project and nationals priorities, policies and strategies Appreciation from national stakeholders with respect to adequacy of project design and Implementation to national realities and existing capacities Level of Involvement of government officials and other partners in the project design process Coherence between needs expressed by national stakeholders and UNDP GEF Criteria Strength of the link between expected results from the project and the needs of relevant stakeholders Degree of involvement and inclusiveness of stakeholders in project design and implementation Project documents National policies and strategies to implement the conventions, or related to environment more generally International convention web sites Project Documents GEF focal areas strategies and documents Expert s Interview reports. Project documents National policies and strategies Key project Partners. Project partners and stakeholders Needs assessment studies Project documents Documents analyses Interviews with project team, RTU, UNDP and other partners Documents analyses GEF website Interviews with UNDP/ RTU and project team Documents analyses Interviews with UNDP/RTU/ and project partners Document analysis Interviews with all relevant stakeholders
Is the project internally coherent in its design? How is the project relevant with respect to other donor supported Activities? Does the project provide relevant lessons and experiences for other similar projects in the future? Implementation? Are there Logical linkages between expected results of the project (log frame) and the project design (in terms of project components, choice of partners, structure, delivery mechanism, scope, budget, use of Resources etc)? Is the length of the project sufficient to achieve Project outcomes? Does the GEF funding support activities and objectives not addressed by other donors? How do GEF funds help to fill gaps (or give additional stimulus) that are necessary but are not covered by other donors? Is there coordination and complementarily between donors Has the experience of the project provided relevant lessons for other future projects targeted at similar objectives? Level of coherence between project Expected results and project design internal logic Level of coherence between project Design and project implementation approach Degree to which program was coherent and complementary to other donor programming nationally and regionally Effectiveness: To what extent have/will the expected outcomes and objectives of the project been/be achieved? Has the project been effective in achieving the expected outcomes and objectives? Has the project been effective in achieving its expected outcomes? See indicators in project document results framework and log frame How have and are risks and risk mitigation being managed? What lessons can be drawn Regarding effectiveness for other similar projects in the future? How well are risks, assumptions and impact drivers being managed? What was the quality of risk mitigation strategies developed? Were these sufficient? Are they institutionalized for future learning and cooperation? Are there clear strategies for risk mitigation related with long term sustainability of the project? What lessons have been learned from the project regarding achievement of outcomes? What changes could have been made (if any) to the design of the project in order to improve the achievement of the project s expected results? Completeness of risk identification and assumptions during project planning and Design Quality of existing information systems in place to identify emerging risks and other issues Quality of risk mitigations strategies developed and followed Program and Project documents Key project stakeholders Documents from other donor supported activities Other donor representatives Project documents Data collected throughout evaluation Project documents Project team and relevant stakeholders Data reported in project reports Project documents UNDP, project team, and relevant stakeholders Data collected throughout the evaluation How effectively have Timely and transparent information on available Track studies MoF, Ministry and or Document analysis.key interviews Documents analyses Interviews with project partners and relevant stakeholders Data analysis Document analysis Interviews Document analysis Interviews Data analysis
funds from the program been transferred to local partners or governments? funds Timely disbursement Correspondence between information on funds, released and received amounts Well defined (and respected) payment triggers Relation to other (government) funds Interviews Document analysis Efficiency: Was the project implemented efficiently, in line with international and national norms and standards? Was project support provided in an efficient way? How efficient are partnership arrangements for the project? Did the project efficiently utilize local capacity in implementation? Was adaptive management used or needed to ensure efficient resource use? Did the project logical framework and work plans and any changes made to them use as management tools during implementation? Were the accounting and financial systems in place adequate for project management and producing accurate and timely financial information? Were progress reports produced accurately, timely and responded to reporting requirements including adaptive management changes? Was project implementation as cost effective as originally proposed (planned vs. actual) Did the leveraging of funds (co financing) happen as planned? Were financial resources utilized efficiently? Could financial resources have been used more efficiently To what extent partnerships/ linkages between institutions/ organizations were encouraged and supported? Which partnerships/linkages were facilitated? Which ones can be considered sustainable? What was the level of efficiency of cooperation and collaboration arrangements? Which methods were successful or not and why? Was an appropriate balance struck between utilization of international expertise as well as local capacity? Did the project taken into account local capacity in design and implementation of the project? Was there an effective collaboration between institutions responsible for implementing the project? Availability and quality of financial and progress reports Timeliness and adequacy of reporting provided Level of discrepancy between planned and utilized financial expenditures Planned vs. actual funds leveraged Cost in view of results achieved compared to costs of similar projects from other organizations Adequacy of project choices in view of existing context, infrastructure and cost Quality of results based management reporting (progress reporting, monitoring and evaluation) Occurrence of change in project design/ implementation approach (i.e. restructuring) when needed to improve project efficiency Specific activities conducted to support the development of cooperative arrangements between partners, Examples of supported partnerships Evidence that particular partnerships/linkages will be sustained Types/quality of partnership cooperation methods utilized Proportion of expertise utilized from international experts compared to national Experts Number/quality of analyses done to assess local capacity potential and absorptive capacity Local Governments, RTU Receiving agents: Local governments Association of NGOs Project documents and evaluations UNDP Project team Project documents and evaluations Project partners and relevant stakeholders Project documents and Evaluations UNDP Beneficiaries Document analysis Key interviews Document Analysis Interviews Document analysis Interviews
What lessons can be drawn Regarding efficiency for similar projects in the future? How effectively has program management implemented the work plans / updated plans to match modified conditions? To what extent have the GEF /UNDP regional office ensured oversight and guidance functions? How well has monitoring and evaluation been linked to the management processes? Are M&E data and reporting used to share / disseminate information and/or to inform strategic decisions? How effective has Technical Advice been in supporting the program? What lessons can be learnt from the project regarding efficiency? How could the project have more efficiently carried out implementation (in terms of management structures and procedures, partnerships arrangements etc )? What changes could have been made (if any) to the project in order to improve its efficiency? Rate of delivery on the annual work plans Achievements against targets (as set out in the ProDoc and in the modified work plans if any) Number of visits Existence of clear mechanisms / instruments to share information and provide feedback Sharing of lessons learnt Responsiveness to requests for TA Existence of baseline data Evidence that an ME systems are set up and updated Evidence that the EMIS system is shared with NGOs Availability of up to date indicators of progress, regular and informative reports Quality, comprehensiveness and timeliness of reporting Degree of use of data from M&E to inform investment decisions Degree of use of data and reports to enhance knowledge base of local and national policy makers Quality of technical reports Responsiveness of reports to program needs Document analysis Interviews Document analysis Interviews Document analysis Interviews Document analysis Interviews Documentary analysis Interviews Data collected throughout evaluation Program reports, Work plans RTUstaff NGO Program reports, RTUstaff, Regional office staff NGO Data sources of M&E unit, reports, RTUstaff, NGO staff Data system used by M&E unit; M&E reports; Interviews with M&E and RTUstaff NGO Program documents Data analysis
Sustainability: How do the objectives of the project relate to the main objective of the AfDB sector and GEF focal area, and to the environment and development priorities at the local, regional and national levels? Has the program been conducive to Evidence of emergence of local sources of revenue to maintain the results of the Document analysis improved efficiency of PES service interventions (user fees) Local budgets and revenue Ministry of Finance delivery and infrastructure Full integration of funds into local budgets statistics maintenance by NGOs and or user National Treasury transfers Unit costs compared to groups? Private public partnerships, municipal bonds other providers How far are NGOs and new community /user groups empowered (legal, technical and economic capacity) to maintain the pilot Infrastructure and services? Evidence of planning, programming, funding and timely implementation of maintenance of infrastructure Evidence of use of local sources of financing to maintain the results of the interventions (e.g. user fees) To what extent is NRM planning and management processes fully embedded (owned) at the LG level? Are NGOs and user groups (legal, Evidence of active involvement of user groups in planning and management? technical and economic capacity) Evidence of significant initiatives taken by NGOs after the intervention empowered to access additional Evidence of LG capacity to tap on other resources: pooled funds, private public funding? partnership, national transfers Is the continued involvement of NGOs in planning and implementation likely? Evidence of local actors engaged in the local management process after the intervention Improved working relations with service providers, institutions and citizens Document analysis Interviews Interviews Budgeting, monitoring & planning document analysis Document analysis Interviews Council, Mayor s office Local Administration Working sessions with NGO and MPA associations of users Council, or Mayor s office Administration Working sessions with associations of users Council, or Mayor s office Administration Communities Planning committees.
Impact Has the project reached its targets and goals, i.e. positive and visible environmental change, improved livelihoods, institutional capacity and learning systems for ERM Has the project reached it targets based on anticipated results? Does original outcomes and targets make sense based on project objectives and overall goals? Impact of ProGeBe Strategic Intervention Line I AfDB subcomponent GEF outcome Preservation of genetic characteristics and improvement of endemic ruminant livestock production and productivity A1 1 Outcome Criteria Indicators Data Collection Methods Data Sources Observations ERL characteristics are conserved and their production and productivity sustainably improved 1/ An increase in the productivity of the three endemic ruminant livestock target species controlling for year effect Mean milk production per cow (MMPC) goal = +40% Mean milk production per goat (MMPG); goal = +40% Annual mortality rate to 1 year* o Gambia (calves; lambs; kids); o Guinee (calves; lambs; kids); o Mali (calves; lambs; kids); o Senegal (calves; lambs; kids). 2/ Number of genetic improvement systems (one for cattle and one for small ruminant) established in each country, based on potential for impact and with sustainability plans. 3/ Proposal: Preference for ERL or not? Numbers of ERL increased nationally more that the average increase. MMPC: > 1.45 kg MMPG > 0.7 kg Mortality rate in % and F/M for calves; lambs; kids Gambia: (16/15; 28/24; 30/31); Guinee (13/8; 18/16; 13/13); Mali (8/9; 21/22; 14/9); Senegal (3/0; 12/5; 3/3) 4 cattle programs 4 sheep programs 4 goat programs Preference at start o For MMP and mortality rates: Document analysis Interviews with livestock owners (guideline B) Most significant changes For GI program Interviews with livestock services and project Focus group discussions (FGD) Sources are not specific for the indicators: Research centers reports Livestock services reports Thesis student studies Reports from expert s structured interviews with beneficiaries Project documents Reports from expert s FGDs with beneficiaries * Changed from Survival rate because an increase of 25% of 0.86 will be higher than 1, which is not possible! So we set as goal to reach: decrease of mortality rates with 25%. Which breed would you prefer to raise: the ERL or another breed? Why? Did you always think so?
ProGeBe Strategic Intervention Line II AfDB subcomponent GEF outcome Improvement of the valorization of endemic ruminant livestock and its products A2 2 Outcome Criteria Indicators * Data Collection Data Sources Observations Methods Commercialization and marketing systems for ERL and livestock products are improved. 1 3/ Analysis of documents 1. Number of ERL producers selling live animals and livestock products over total number of ERL producers at site level (+15% at project end). 2. Percentage of ERL producers accessing selected financial and market services (credit, market infrastructure, banking services) over total number of ERL producers at site level (+50% at project end). 3. Value of ERL products sold over total value of ruminant livestock products (ERL and non ERL) sold (+15% at project end). 4. Proposal: The real market price of ERL increased 10%. The demand and price for milk of ERL increased 10%. 1/ Gambia: Cattle= 16 % Sheep= 17% Goats= 19% Guinea: data not available in baseline Mali: data not available in baseline Senegal: Cattle= 46 % Sheep= 21% Goats= 20% 2/ Gambia: data not available in baseline, but where do data in logframe come from: Cattle= 64% Sheep=53% Goats=56.7% Guinea: baseline not available Mali: baseline not available Senegal: baseline not available. 3/ Gambia: 31.3% Guinea: baseline not available Mali: baseline not available Senegal: baseline not available 4/ (Market prices) increased 10% (more than inflation) * The country Baseline studios did not collect (or not calculate) the data for all criteria mentioned in the LogFrame! 4/ Analysis of documents Interviews of beneficiaries Focus group discussions 1/ Report of project or livestock services. 2/ Report of project or livestock services 3/ Report of livestock services 4/ Reports of livestock services and of project Reports from expert s structured interviews with beneficiaries. Reports from expert s focus groups discussion with beneficiaries. Reports from expert s structured interviews with dairy plants. 4/ Question for FGD: Access to credit NRM Please describe the historical development of ERL market prices? ProGeBe Strategic Intervention Line III AfDB subcomponent GEF outcome Sustainable management of endemic ruminant livestock ecosystems B 3 Outcome Criteria Indicators * Data Collection Data Sources Observations Methods Natural resources in project pilot sites conserved and sustainably managed for the benefit of endemic ruminant livestock, ecosystem services, and human livelihoods 1. Percentage of producers adopting / participating in the community based sustainable management of NR practices promoted by the project as a share of the total number of producers. 2. Number of ERL natural habitat zones (including protected areas identified, delimited and conserved in each site under community based and integrated approach to the management of ERL and its habitats. 3. Number and frequency of uncontrolled rangeland fires in project pilot sites. 1/ 60% at the end of the project 2/ At least one (1) natural habitat zone established at each project pilot site at mid term 3/ 50 % decrease of uncontrolled rangeland fires at the end of the project. 1 / Analysis of documents 2 and 3/ Analysis of documents and expert s focus groups discussion 1 3/ Community surveys 2 3/ Report of project or livestock services 3/. Reports from expert s focus groups discussion with beneficiaries.
ProGeBe Strategic Intervention Line IV AfDB subcomponent GEF outcome Legal, policy and institutional frameworks. B and Crosscutting 4 Outcomes Criteria Indicators * Data Collection Methods Data Sources Observations Legal, policy and institutional frameworks established at the local, national, and sub regional level for in situ conservation of endemic ruminant livestock. 1. Increases in the scores of the Policy and Regulatory Screening Tool to be applied in 2010, mid term and end of the project, showing that the enabling environment for the conservation, production, and marketing of ERL is improving in the four targeted countries 2. Increase in the budget allocations for the preservation of endemic ruminant livestock and their habitat in the central and local government. 1/ To be clarifed and subsequently determined 2/ Government engagement increase by 20% at the end of the project. 1/ Analysis of documents 2/ Analysis of national, provincial and district budget or plans. Interviews of local stakeholders and boundary partners, 1/ Local and central government s plans and progammes 2/ National, provincial and district budget or plans Reports from expert s structured interviews with local government en boundary partners.. ProGeBe Strategic Intervention Line V A AfDB subcomponent GEF outcome Cooperation, knowledge management, exchanges and coordination Crosscutting 5 Outcomes Criteria Indicators * Data Collection Methods Data Sources Observations A sub regional system for cooperation, information exchange, and coordinated support for the sustainable management of endemic ruminant livestock is established and operational 1/ The value added of platforms/networks set up by the project for information exchanges, cross fertilization and coordinating purpose on genetic, NRM management, animal production and marketing, at local, national, and regional levels, as independently assessed through evaluation. 2/ Numbers and types of stakeholders participating in information sharing platforms at local, national and regional level. 1/ The use, by the receivers, of the information. 2/ Government engagement increase by 20% at the end of the project. 1/ Assessment of the use of the information by the users. 2/ Assessment of the users / participants of the platforms. Interviews of beneficiaries. 1/ Report of an independent assessment 2/ Report of an independent assessment Reports from expert s structured interviews with local stakeholders and boundary partners The Report of an independent assessment may not be available