Historique

1. Introduction

In accordance with the United Nations Development Programme (UNDP) and the Global Environment Facility (GEF)’s Monitoring and Evaluation (M&E) policies and procedures, all full-sized UNDP- supported GEF-financed projects are required to undergo a Terminal Evaluation (TE) towards the end of the project. This Terms of Reference (ToR) sets out the expectations for the TE of the project titled Accelerating the Development and Commercialization of Fuel Cell Vehicles in China (PIMS #5349) implemented through the Ministry of Science and Technology of People’s Republic of China/UNDP China Office. The project started on the Project Document signature date (15 August 2016) and is in its 5th year of implementation plus a 12-month extension till end August 2021. The TE process must follow the guidance outlined in the document ‘Guidance For Conducting Terminal Evaluations of UNDP-Supported, GEF-Financed Projects’.

2.Project Description 

The Accelerating the Development and Commercialization of Fuel Cell Vehicles in China Project aims to facilitate commercialization of fuel cell vehicles (FCVs) in China. It will achieve this through a multi-pronged strategy that will enable China to (a) “leapfrog” in its FCV durability/performance improvements and cost reductions far beyond what would be achieved in the baseline scenario and (b) get many more FCVs on the road by end of project than would occur in the baseline scenario.

The strategy will consist of components covering the areas of: (1) FCV and FC technology improvement/cost reduction (raising technical abilities and international sourcing connections of China’s FCV manufacturers, raising technical abilities of its FCV component manufacturers, and demonstrating 109 FCVs across 4 demo cities); (2) hydrogen production and hydrogen refuelling stations (introducing in China renewable energy-based hydrogen production of substantial scale and demonstrating at least 4 hydrogen refuelling stations with varied business models); (3) policy (covering national FCV Roadmap, standards and certification, expedited approval processes, and stabilized and expanded incentive policies, including two policy pilots); (4) awareness and information dissemination (addressing the general public, government officials, etc. and ensuring replication); and (5) capacity building (covering FCV and hydrogen refuelling station O&M and the financial sector’s knowledge of and ability to assess investments and loans in FCV-related areas).

3.TE Purpose

The TE report will assess the achievement of project results, realization of the project outcomes, and achievement of the project objective against what was expected to be achieved and draw lessons that not only improve the sustainability of benefits from this project, and aid in the overall enhancement of UNDP programming. The TE report promotes accountability and transparency and assesses the extent of project accomplishments.

Devoirs et responsabilités

TE Approach & Methodology

The TE must provide evidence-based information that is credible, reliable and useful such as tangible outputs that the project is designed to deliver, as well as documents (e.g., reports, databases, etc.) for verifying and confirming the achievement of end-of-project targets, and for confirming the actual amounts of project funding (GEF and non-GEF) that were spent. The TE task force is comprised of 1 international and 1 national consultant, with the international consultant taking the lead in organizing the evaluation and drafting the report. The national consultant is expected to fully support the international consultant for completing the evaluation.

The TE team will review all relevant sources of information including documents prepared during the preparation phase (i.e. PIF, UNDP Social and Environmental Screening Procedure/SESP) the Project Document, project reports including annual PIRs, Mid-Term Evaluation (MTR) report, Project Appraisal Committee meeting minutes, project budget revisions, relevant lessons learned/reports, national strategic and legal documents, and any other materials that the team considers useful for this evidence-based evaluation. The TE team will review the baseline and midterm GEF focal area Core Indicators/Tracking Tools submitted to the GEF at the CEO endorsement and midterm stages and the terminal Core Indicators/Tracking Tools that must be completed before the TE field mission begins.

The TE team is expected to follow a participatory and consultative approach ensuring close engagement with the Project Team, government counterparts, Implementing Partners, the UNDP Country Office(s), the Regional Technical Advisors, direct beneficiaries, and other stakeholders.

Engagement of stakeholders is vital to a successful TE. Stakeholder involvement should include interviews with stakeholders who have project responsibilities, including but not limited to executing agencies, senior officials and task team/component leaders, key experts and consultants in the subject area, Project Board, project beneficiaries, academia, local government and CSOs, etc. Additionally, the TE team is expected to conduct field missions to the project demo cities such as Beijing, Shanghai, Zhengzhou, etc. including the following project sites i.e. bus operations companies, hydrogen refueling stations, vehicle OEMs etc.

The specific design and methodology for the TE should emerge from consultations between the TE team and the above-mentioned parties regarding what is appropriate and feasible for meeting the TE purpose and objectives and answering the evaluation questions, given limitations of budget, time and data. The TE team must, however, use gender-responsive methodologies and tools and ensure that gender equality and women’s empowerment, as well as other cross-cutting issues and SDGs are incorporated into the TE report.

The final methodological approach including interview schedule, field visits and data to be used in the evaluation should be clearly outlined in the inception report and be fully discussed and agreed between UNDP, stakeholders and the TE team. The methodologies of calculation needed for reporting against project indicators, if not specified clearly in the ProDoc, shall be determined jointly the UNDP Office(s), the IP, and expert(s) in this field.

The final TE report should describe the full TE approach taken and the rationale for the approach making explicit the underlying assumptions, challenges, strengths and weaknesses about the methods and approach of the evaluation.

Detailed Scope of the TE

The TE will assess project performance against expectations set out in the project’s Logical Framework/Results Framework (see TOR Annex A). The TE will assess results according to the criteria outlined in the Guidance for TEs of UNDP-supported GEF-financed Projects.

The Findings section of the TE report will cover the topics listed below. A full outline of the TE report’s content is provided in ToR Annex C. The asterisk “(*)” indicates criteria for which a rating is required.

Findings

Project Design/Formulation

• National priorities and country driven-ness
• Theory of Change, if applicable
• Gender equality and women’s empowerment
• Social and Environmental Safeguards
• Analysis of Results Framework: project logic and strategy, indicators
• Assumptions and Risks
• Lessons from other relevant projects (e.g. same focal area) incorporated into project design
• Planned stakeholder participation
• Linkages between project and other interventions within the sector
• Management arrangements

Project Implementation

• Adaptive management (changes to the project design and project outputs during implementation)
• Actual stakeholder participation and partnership arrangements
• Project Finance and Co-finance
• Monitoring & Evaluation: design at entry (*), implementation (*), and overall assessment of M&E (*)
• Implementing Agency (UNDP) (*) and Executing Agency (*), overall project
• oversight/implementation and execution (*)
• Risk Management, including Social and Environmental Standards

Project Results

• Assess the achievement of outcomes against indicators by reporting on the level of progress for each objective and outcome indicator at the time of the TE and noting final achievements
• Relevance (*), Effectiveness (*), Efficiency (*) and overall project outcome (*)
• Sustainability: financial (*), socio-political (*), institutional framework and governance (*),
• environmental (*), overall likelihood of sustainability (*)
• Country ownership
• Gender equality and women’s empowerment
• Cross-cutting issues (poverty alleviation, improved governance, climate change mitigation and
• adaptation, disaster prevention and recovery, human rights, capacity development, South-South
• cooperation, knowledge management, volunteerism, etc., as relevant)
• GEF Additionality
• Catalytic Role / Replication Effect
• Progress to impact

Main Findings, Conclusions, Recommendations and Lessons Learned

• The TE team will include a summary of the main findings of the TE report. Findings should be presented as statements of fact that are based on analysis of the data at the end of the TE mission.
• The section on conclusions will be written in light of the findings. Conclusions should be comprehensive and balanced statements that are well substantiated by evidence and logically connected to the TE findings. They should highlight the strengths, weaknesses and results of the project, respond to key evaluation questions and provide insights into the identification of and/or solutions to important problems or issues pertinent to project beneficiaries, UNDP and the GEF, including issues in relation to gender equality and women’s empowerment.
• Recommendations should provide concrete, practical, feasible and targeted recommendations directed to the intended users of the evaluation about what actions to take and decisions to make. The recommendations should be specifically supported by the evidence and linked to the findings and conclusions around key questions addressed by the evaluation. A recommendation table should be put in the report’s executive summary. While it is not obligatory to give recommendations to each project output, the TE consultant/team is encouraged to identify and assess recommendations to the Project Team but should make no more than 15 recommendations in total.
• The TE report should also include lessons that can be taken from the evaluation, including best and worst practices in addressing issues relating to relevance, performance and success that can provide knowledge gained from the particular circumstance (programmatic and evaluation methods used, partnerships, financial leveraging, etc.) that are applicable to other GEF and UNDP interventions. When possible, the TE team should include examples of good practices in project design and implementation.
• It is important for the conclusions, recommendations and lessons learned of the TE report to include results related to gender equality and empowerment of women.

The TE report will include an Evaluation Ratings Table, as shown in the ToR Annex.

Compétences

Expected Outputs and Deliverables

The TE task force shall prepare and submit:

• TE Inception Report: TE team clarifies objectives and methods of the TE no later than 2 weeks before the TE mission. TE team submits the Inception Report to the Commissioning Unit and project management. Approximate due date: 13 Mar 2021
• Presentation: TE team presents initial findings to project management and the Commissioning Unit at the end of the TE mission. Approximate due date: 12 Apr 2021
• Draft TE Report: TE team submits full draft report with annexes within 3 weeks of the end of the TE mission. Approximate due date: 7 May 2021
• Final TE Report* and Audit Trail: TE team submits revised report, with Audit Trail detailing how all received comments have (and have not) been addressed in the final TE report, to the Commissioning Unit within 1 week of receiving UNDP comments on draft. Approximate due date: 15 May 2021

*The final TE report must be in English. If applicable, the Commissioning Unit may choose to arrange for a translation of the report into a language more widely shared by national stakeholders.

All final TE reports will be quality assessed by the UNDP Independent Evaluation Office (IEO). Details of the IEO’s quality assessment of decentralized evaluations can be found in Section 6 of the UNDP Evaluation Guidelines.

TE Arrangements

The principal responsibility for managing this TE resides with the Commissioning Unit. The Commissioning Unit for this project’s TE is UNDP China.

The Commissioning Unit will contract the consultants and ensure the timely provision of per diems and travel arrangements within the country for the TE team. The Project Team will be responsible for liaising with the TE team to provide all relevant documents, set up stakeholder interviews, and arrange field visits.

Duration of the Work

The total duration of the TE will be approximately 20 working days over a time period of 16 weeks starting 9 Mar 2021 and shall not exceed five months from when the TE team is hired. The tentative TE timeframe is as follows:

• 04 Mar 2021: Application closes
• 5 Mar 2021: Selection of TE Team
• 9 Mar 2021 (0.5 working day): Prep the TE Team (handover of project documents)
• 10-11 Mar 2021 (2 working days within the given period): Document review and preparing TE Inception Report
• 12-13 Mar 2021 (1 working day within the given period): Finalization and Validation of TE Inception Report
• 22 Mar - 12 Apr 2021 (5.5 working days within the given period): TE mission: stakeholder meetings, interviews, field visits; on the last day of the TE mission, a mission wrap-up meeting & presentation of initial findings should be conducted
• 13 Apr - 7 May 2021 (9 working days within the given period): Preparing draft TE report
• 7 May 2021: Circulation of draft TE report for comments
• 7 - 30 May 2021 (2 working days within the given period): Incorporation of comments on draft TE report into Audit Trail & finalization of TE report
• 10 May 2021: Preparation & Issue of Management Response by UNDP China
• 15 May 2021: (optional) Concluding Stakeholder Workshop; Expected date of full TE completion

As a mission in China is required for the TE and in light of the concurrent pandemic, candidates that will be already based in China with disease control measures (i.e. mandatory quarantine, nucleic test etc.) completed close to the mission date will have a strong advantage in the selection process. Please make sure to clarify the relevant information in your application and technical proposals.

The date start of contract is 9 Mar 2021.

Duty Station

All related travel expenses will be covered and will be reimbursed as per UNDP rules and regulations upon submission of an F-10 claim form and supporting documents.

The consultant’s duty station/location for the contract duration is mainly home based with mission to Beijing and field visits to the pilot cities. Given that it may be impractical to conduct missions to all pilot cities (Beijing, Shanghai, Zhengzhou, Foshan, Yancheng, Zhangjiakou and Changshu), the TE task force and UNDP China shall jointly decide and select cities for the mission.

Travel:

• A mission in China is required during the TE; in light of the concurrent pandemic, candidates who  are currently based in China with pandemic control measures (i.e. mandatory quarantine, nucleic test etc.) or have the quarantine requirements completed close to the mission date will be given an advantage in the selection process. Please make sure to clarify the relevant information in your application and technical proposals.
• The Basic Security in the Field II and Advanced Security in the Field (BSAFE) courses must be successfully completed prior to commencement of travel;
• Consultants are responsible for ensuring they have vaccinations/inoculations when travelling to certain countries, as designated by the UN Medical Director.
• Consultants are required to comply with the UN security directives set forth under https://dss.un.org/dssweb/.NOTE: This is now consolidated in BSAFE.

Qualifications et expériences requises

The TE team will consist of two independent consultants that will conduct the TE - one team leader (with international experience and exposure to projects and evaluations in other regions) and one team expert from China.  The team leader will be mainly responsible for the overall design and writing of the TE report. The team expert will support the team leader in drafting the TE report, provide local industry insights, conduct researches in Chinese, work with the Project Team in developing the TE itinerary, etc.

As requested in the Guidance for Conducting Terminal Evaluations of UNDP-supported, GEF-financed Projects, the evaluator(s) cannot have participated in the project preparation, formulation and/or implementation (including the writing of the project document), must not have conducted this project’s Mid-Term Review and should not have a conflict of interest with the project’s related activities.

The selection of consultants will be aimed at maximizing the overall “team” qualities in the following areas:

• Recent experience with result-based management evaluation methodologies (5%);
• Experience applying SMART indicators and reconstructing or validating baseline scenarios (5%);
• Competence in adaptive management, as applied to energy, decarbonization, especially in relation to the automotive industry; fuel cell related technology or commercialization expertise a strong asset (10%);
• Experience working with the GEF or GEF-evaluations (20%);
• Experience working in Asia, especially in China (5%);
• Work experience in relevant technical areas for at least 5 years (10%);
• Demonstrated understanding of issues related to gender sensitive evaluation and analysis (10%);.
• Excellent communication skills (10%);
• Demonstrable analytical skills (10%);
• Project evaluation/review experiences within United Nations system will be considered an asset (10%);
• A Master’s degree in engineering, environmental management, industrial development, or other closely related field (5%).

Evaluator Ethics:

The TE team will be held to the highest ethical standards and is required to sign a code of conduct upon acceptance of the assignment. This evaluation will be conducted in accordance with the principles outlined in the UNEG ‘Ethical Guidelines for Evaluation’. The evaluator must safeguard the rights and confidentiality of information providers, interviewees and stakeholders through measures to ensure compliance with legal and other relevant codes governing collection of data and reporting on data. The evaluator must also ensure security of collected information before and after the evaluation and protocols to ensure anonymity and confidentiality of sources of information where that is expected. The information knowledge and data gathered in the evaluation process must also be solely used for the evaluation and not for other uses without the express authorization of UNDP and partners.

 Payment Schedule

• 20% payment upon satisfactory delivery of the final TE Inception Report and approval by the Commissioning Unit
• 80% payment upon satisfactory delivery of the final TE report and approval by the Commissioning Unit and RTA (via signatures on the TE Report Clearance Form) and delivery of completed TE Audit Trail

Criteria for issuing the final payment of 80%

• The final TE report includes all requirements outlined in the TE TOR and is in accordance with the TE guidance.
• The final TE report is clearly written, logically organized, and is specific for this project (i.e. text has not been cut & pasted from other MTR reports).

The Audit Trail includes responses to and justification for each comment listed.

1. Recommended Presentation of Proposal

1. Completed Letter of Confirmation of Interest and Availability using the template provided by UNDP;
2. Personal CV or a P11 Personal History form, indicating all past experience from similar projects, as well as the contact details (email and telephone number) of the Candidate and at least three (3) professional references;
3. Brief description of approach to work/technical proposal of why the individual considers him/herself as the most suitable for the assignment, and a proposed methodology on how they will approach and complete the assignment; (max 1 page)
4. Financial Proposal that indicates the all-inclusive fixed total contract price and all other travel related costs (such as flight ticket, per diem, etc.), supported by a breakdown of costs, as per template attached to the Letter of Confirmation of Interest template. If an applicant is employed by an organization/company/institution, and he/she expects his/her employer to charge a management fee in the process of releasing him/her to UNDP under Reimbursable Loan Agreement (RLA), the applicant must indicate at this point, and ensure that all such costs are duly incorporated in the financial proposal submitted to UNDP.

All application materials should complete the application on the UNDP website. Incomplete applications will be excluded from further consideration.

1. Scope of Price Proposal and Schedule of Payments

Financial Proposal:

• Financial proposals must be “all inclusive” and expressed in a lump-sum for the total duration of the contract. The term “all inclusive” implies all cost (professional fees, travel costs, living allowances etc.);
• For duty travels, all living allowances required to perform the demands of the ToR must be incorporated in the financial proposal, whether the fees are expressed as daily fees or lump sum amount;

The lump sum is fixed regardless of changes in the cost components.

1. Criteria for Selection of the Best Offer

The award of the contract will be made to the Individual Consultant who has obtained the highest Combined Score and has accepted UNDP’s General Terms and Conditions.  Only those applications which are responsive and compliant will be evaluated. The offers will be evaluated using the “Combined Scoring method” where:

1. The educational background and experience on similar assignments will be weighted a max. of 70%;
2. The price proposal will weigh as 30% of the total scoring.

1. Annexes to the TE ToR

• ToR Annex A: Project Logical/Results Framework
• ToR Annex B: Project Information Package to be reviewed by TE team
• ToR Annex C: Content of the TE report
• ToR Annex D: Evaluation Criteria Matrix template
• ToR Annex E: UNEG Code of Conduct for Evaluators
• ToR Annex F: TE Rating Scales and TE Ratings Table
• ToR Annex G: TE Report Clearance Form
• ToR Annex H: TE Audit Trail template

ToR Annex A: Project Logical/Results Framework

ToR Annex B: Project Information Package to be reviewed by TE team

ToR Annex C: Content of the TE report

1. Title page

• Tile of UNDP-supported GEF-financed project
• UNDP PIMS ID and GEF ID
• TE timeframe and date of final TE report
• Region and countries included in the project
• GEF Focal Area/Strategic Program
• Executing Agency, Implementing partner and other project partners
• TE Team members

1. Acknowledgements
2. Table of Contents
3. Acronyms and Abbreviations

1. Executive Summary (3-4 pages)

• Project Information Table
• Project Description (brief)
• Evaluation Ratings Table
• Concise summary of findings, conclusions and lessons learned
• Recommendations summary table

1. Introduction (2-3 pages)

• Purpose and objective of the TE
• Scope
• Methodology
• Data Collection & Analysis
• Ethics
• Limitations to the evaluation
• Structure of the TE report

1. Project Description (3-5 pages)

• Project start and duration, including milestones
• Development context: environmental, socio-economic, institutional, and policy factors relevant to the project objective and scope
• Problems that the project sought to address: threats and barriers targeted
• Immediate and development objectives of the project
• Expected results
• Main stakeholders: summary list
• Theory of Change

1. Findings

(in addition to a descriptive assessment, all criteria marked with (*) must be given a rating[5])

4.1 Project Design/Formulation

• Analysis of Results Framework: project logic and strategy, indicators
• Assumptions and Risks
• Lessons from other relevant projects (e.g. same focal area) incorporated into project design
• Planned stakeholder participation
• Linkages between project and other interventions within the sector
  1. Project Implementation

• Adaptive management (changes to the project design and project outputs during implementation)
• Actual stakeholder participation and partnership arrangements
• Project Finance and Co-finance
• Monitoring & Evaluation: design at entry (*), implementation (*), and overall assessment of M&E (*)
• UNDP implementation/oversight (*) and Implementing Partner execution (*), overall project implementation/execution (*), coordination, and operational issues
  1. Project Results

• Progress towards objective and expected outcomes (*)
• Relevance (*)
• Effectiveness (*)
• Efficiency (*)
• Overall Outcome (*)
• Country ownership
• Gender
• Other Cross-cutting Issues
• Social and Environmental Standards
• Sustainability: financial (*), socio-economic (*), institutional framework and governance (*), environmental (*), and overall likelihood (*)
• Country Ownership
• Gender equality and women’s empowerment
• Cross-cutting Issues
• GEF Additionality
• Catalytic Role / Replication Effect
• Progress to Impact

1. Main Findings, Conclusions, Recommendations & Lessons

• Main Findings
• Conclusions
• Recommendations
• Lessons Learned

1. Annexes

• TE ToR (excluding ToR annexes)
• TE Mission itinerary
• List of persons interviewed
• List of documents reviewed
• Summary of field visits
• Evaluation Question Matrix (evaluation criteria with key questions, indicators, sources of data, and methodology)
• Questionnaire used and summary of results
• Co-financing tables (if not include in body of report)
• TE Rating scales
• Signed Evaluation Consultant Agreement form
• Signed UNEG Code of Conduct form
• Signed TE Report Clearance form
• Annexed in a separate file: TE Audit Trail
• Annexed in a separate file: relevant terminal GEF/LDCF/SCCF Core Indicators or Tracking Tools, as applicable

ToR Annex D: Evaluation Criteria Matrix template

ToR Annex E: UNEG Code of Conduct for Evaluators

Independence entails the ability to evaluate without undue influence or pressure by any party (including the hiring unit) and providing evaluators with free access to information on the evaluation subject.  Independence provides legitimacy to and ensures an objective perspective on evaluations. An independent evaluation reduces the potential for conflicts of interest which might arise with self-reported ratings by those involved in the management of the project being evaluated.  Independence is one of ten general principles for evaluations (together with internationally agreed principles, goals and targets: utility, credibility, impartiality, ethics, transparency, human rights and gender equality, national evaluation capacities, and professionalism).

ToR Annex F: TE Rating Scales

ToR Annex G: TE Report Clearance Form

ToR Annex H: TE Audit Trail

To the comments received on (date) from the Terminal Evaluation of COMPREHENSIVE REDUCTION AND ELIMINATION OF PERSISTENT ORGANIC POLLUTANTS IN PAKISTAN” (UNDP Project PIMS # 4600)

The following comments were provided to the draft TE report; they are referenced by institution/organization (do not include the commentator’s name) and track change comment number (“#” column):

[1] Emission reductions of 132,707 tons CO2 by EOP are a combination of direct incremental net ERs (for 109 FCVs and 4 renewable energy based hydrogen production units) and indirect ERs (assuming total vehicles by EOP are 4,000 including original 109 and assuming an additional 12 renewable energy based hydrogen production units by EOP). Direct incremental net ERs total 15,287 tons, of which 9,365 tons are due to the 109 FCVs operating for 3.2 years (with baseline scenario subtracted out) and 5,922 tons are due to the four renewable energy based hydrogen production facilities operating for two years before EOP (with double counting for the portion of hydrogen used in the demo FCVs subtracted out). Indirect ERs by EOP total 117,420 tons, of which 108,537 tons are due to additional FCVs (891 of which come online by start of year 3 and another 1500 of which come online by start of year 4) and 8,883 tons are due to an additional 12 renewable energy based hydrogen production facilities (which come online by start of year four).

[2] Objective (Atlas output) monitored quarterly ERBM  and annually in APR/PIR

[3] Projection based on production volume of 500 units for buses and 5,000 units for cars, vans, and trucks

[4] Projection based on production volume of 500 units for buses and 5,000 units for cars, vans, and trucks

[5] See ToR Annex F for rating scales.