Background

Terms of Reference for Individual Contract to conduct Evaluation of the Project “Promotion of Renewable Energy and Energy Efficient Technologies in the Building Sector of Bhutan: Enabling Environment”

POST TITLE:          Evaluation Team to conduct Evaluation of the Project “Promotion of Renewable Energy and Energy Efficient Technologies in the Building Sector of Bhutan: Enabling Environment” (International Team leader)

CONTRACTING AGENCY:         UNDP BHUTAN

COORDINATING AGENCY:       Gross National Happiness Commission

COUNTRY OF ASSIGNMENT:    Bhutan

EXPECTED DURATION:              25 working Day w.e.f  May 31 to 28 June 2019

1. Introduction/Background

The project titled “Promotion of Renewable and Energy Efficient Technologies in the Building Sector of Bhutan - Enabling Environment” is implemented by the Department of Renewable Energy (DRE) under the Ministry of Economic Affairs, and Department of Engineering Services (DES) under the Ministry of Works and Human Settlement. The project is being implemented with the objective to establish a solid foundation and an enabling framework for improving the energy performance of buildings and use of solar thermal energy. The current project was officially signed on 25 May 2017 with a grant of EUR 660,900. The in-kind contribution of the Royal Government of Bhutan is estimated at 180,000 EUR which covered the costs for the project management staff, counterpart staff and procurement of land. The project started in May 2017 with expected completion by June 2019.

There are two main outcomes of project namely to improve framework conditions for wide scale market introduction of solar thermal (ST) energy technologies, and to improve energy performance of buildings in the long term. Activities towards achievement of the first outcome are being implemented by Department of Renewable Energy, Ministry of Economic Affairs while the latter by the Department of Engineering Services, Ministry of Works and Human Settlement.

The Project Focus Areas:

Outcome 1: The outcome focuses on identification of an appropriate range of technologies for use of solar thermal energy, and improvement of capacities for understanding the technical, social, gender and environmental problems, risks and possible uses of solar thermal technology and its market introduction. The outcome also focused on the improvement of the framework conditions (BBR, definition of subsidy scheme, training of skilled workers) for wide scale market introduction. The major outputs under the outcome are enhancing institutional capacity of DRE on solar thermal technologies, conducting research on best appropriate technologies, enhanced operation and maintenance skills of DRE through refurbishment of existing systems, enhancing preparation measures for piloting of subsidy scheme, and improving dissemination and institutionalization of know-how.

Outcome 2: The outcome focuses on the creation of a strong basis for long term improvement of the energy performance of buildings in the areas of energy performance baseline of buildings; legal framework conditions; scientific, theoretical and practical knowledge; and strengthening human resource capacities to develop and implement improved rules, regulations and guidelines for energy efficient design and construction of buildings. The main outputs under the program are to strengthen institutional capacity to sustain EE construction technology, establish energy baseline for buildings, conduct adaptive research for optimization of alternative EE material and technology transfer, introduce mechanization, improvement and demonstration of traditional rammed earth construction technology in rural areas as energy efficient and sustainable technology, and augment long term EE intervention through update of Guidelines, Rules and Regulations

The project document foresees an independent external final evaluation of the project in the last quarter of project implementation.  

1. Objective

The main objectives of the evaluation are as follows:

1. To assess how effective and efficient the project has been implemented
2. To assess what has been the impact and sustainability of the interventions
3. To assess how the crosscutting issues of gender, social and environmental issues have been addressed in the implementation;
4. To identify challenges, good practice and lessons

Focus/Scope

The evaluation will cover the period from May 2017 to March 2019. The evaluation will use the OECD DAC criteria on relevance, effectiveness, efficiency, impact and sustainability to evaluation the project implementation and performance.

The evaluation will take place in Thimphu, Bhutan and nearby project locations where the implementing partners’ offices are located. It will cover implementing partners of the project namely, the Department of Renewable Energy and Department of Engineering Services and target groups.

1. Evaluation Questions

With regard to the evaluation of the project, the following questions shall be analyzed and answered:

Relevance

1. How relevant are project interventions in meeting the intended outcomes of the project?

Effectiveness

1. To what extent has the project achieved the two outcomes or will be likely to achieve them? What were the major factors influencing the achievement or non-achievement of the outcomes?
2. What are the major outputs and what were the major factors influencing the achievement or non-achievement of the outputs under each of the two outcomes?
3. To what extent can intervention logic be identified and, if so, has it worked?
4. Has the social standard appraisal, or gender, or environment report been taken into account or not, and what implications has this had in terms of effectiveness?
5. To what extent were the international and national technical assistance (TA) embedded and accountable?
6. How has project stakeholder’s cooperation effected the achievement of results and outcomes?
7. To assess whether the approach of combining two projects on one overall project proved to be effective? Were there any benefits in this approach or have there been additional transaction costs?

Efficiency

1. To what extent were all equipment purchased and used as planned under this project?
2. Were the project’s resources especially personnel resources including international TA used in the most efficient way?
3. To assess whether the approach of combining two projects on one overall project proved to be efficient? Were there any benefits in this approach or have there been additional transaction costs?
4. Have any issues emerged, if so which ones and why what extent has the project already achieved the two outcomes or will be likely to achieve them? What were the major factors influencing the achievement or non-achievement of the outcomes?

Impact

1. How many stakeholders including individuals and institutions have already benefited from the project (immediate impact)? What has changed for whom?
2. What is the impact, if any, in terms of capacity building?
3. Which positive and/or negative effects/impacts in terms of gender, environment and social standards can possibly be attributed to the project? How were the recommendations stemming from the environment, gender and social assessments addressed?

Sustainability

1. How should the implementing partners ensure or strengthen capacity to sustain the activities supported or to sustain achievements?
2. What were the major factors which influenced the achievement or non-achievement of sustainability of the project?

1. Approach and Methods

The evaluation must provide evidence-based information that is credible, reliable and useful. The evaluation team shall use a combination of quantitative and qualitative methods including desk-study of available project documents, progress reports and study reports, analyze the intervention logic and its assumptions; use key informant interviews and focus group discussions from the field visits where target groups are located.

Based on the requirements of this ToR the evaluator shall prepare the evaluation design matrix in which the proposed methods shall be presented. The findings and recommendations in the draft and final reports must be structured around the evaluation questions.

Currently, total of  working days has been estimated for the evaluation. The evaluation will include the following steps:

Procurement process: UNDP will carry out the procurement process and select the eligible team of evaluators as per the required criteria.

Contract Signing: Contract agreement will be signed followed by a discussion on the evaluation. Relevant documents, including data and information, will be provided to the evaluator.

Desk Study: The evaluator will review and analyze the available documents and accordingly, prepare the evaluation design matrix.

Inception Report:  In this report, the evaluators will present the proposed evaluation design matrix to the Reference Group, including criteria for field visits and interviews and estimated number of working days per phase. Here, the evaluator will share on how data will be collected, processed and analyzed. The Reference Group shall provide feedback which will have to be incorporated into the Inception Report.

Field Visit: A field visit shall be scheduled in the selected project areas to collect primary data and conduct interviews of the relevant stakeholders. All data collected will have to be disaggregated by sex, age and disability, if possible.

Submission of draft Evaluation Report: The team leader will submit the draft report to the Reference Group for further comments and feedback.

Submission/Presentation of Final Evaluation Report:  After incorporating comments from the relevant stakeholders, the final report will be submitted to the Reference Group and make a final presentation on the findings and recommendations. The final report should not exceed a length of 50 pages, plus annexes.

Reference Group

A Reference Group comprising members of the Project Steering Committee (PSC) will be formed to provide feedback on the draft reports to the evaluators to ensure the quality and standard of the evaluation process. Members of the PSC are: Director of Gross National Happiness Commission, Directors of Department of Engineering Services and Department of Renewable Energy, Head of the Austrian Coordination Office, and the Chief Planning Officer of the Development Cooperation Division of GNHC. In addition, a relevant official from Research and Evaluation Division, GNHC, and ADA (in Bhutan) will be part of reference group.

1. Evaluation Team

The evaluation team will be composed of: senior evaluator (team leader, international), a sector expert (energy, international) and a national evaluator.

Team Leader

Education:

• A post-secondary/advanced degree (Masters level or higher) in environmental sciences, evaluation, or a related subject

Experience:

• A minimum of five years of proven work experience and expertise in the field of evaluation (sample of evaluation report produced should be annexed to offer)
• Experience in evaluation (experience with EU/UNDP and/or GEF-financed projects is an advantage)
• Proven knowledge of Bhutan with focus on topics such as renewable energy and energy efficiency.
• Gender balance in the team is an asset
• Working Experience in national context and be culturally sensitive
• Experience in Project Cycle Management
• Experience and expertise in evaluating cross-cutting issues
• Excellent oral and written English skills
• Sound MS Office and IT skills

The consultants (team leader, energy expert, national team member) must not have been involved in the design, implementation or monitoring of ADC/ADA funded projects on Renewable Energy and Energy Efficient Technologies in Bhutan or other ADA activities related to the scope of this project.

Duty Station

Home based with mission travel to Bhutan

1. Duration of Assignment

Duration:  May 31, 2019 - June 28, 2019

1. Deliverables and Milestones

The evaluation team will submit the following reports:

• Draft and final inception report.
• Draft evaluation report
• Final evaluation including results assessment form

* When submitting the final evaluation report, the evaluator is required also to provide an 'audit trail', detailing how all received comments have (and have not) been addressed in the final evaluation report. See Annex B for an audit trail template.

1. Payment Terms

The payment will be made based on the deliverables of assignment in the following manner: The Payment schedule shall be as follows:

1. Institutional arrangements

The consultants shall work in close consultations with the Reference Group. The team leader shall brief the focal officer in GNHC. The Experts are expected to submit reports as required and agreed in the work plan and time schedule.

The consultant is responsible for its own logistical arrangements such as local transport and accommodation. Any other matters related to visa and facilitation of the meetings shall be arranged by the GNHC. However, the evaluators shall independently meet with the stakeholders.

1. Expected Outcome

The consultant will submit the following reports:

• an inception report (10-15 pages without annexes),
• a final draft evaluation report (about 25-30 pages without annexes), including a draft executive summary and the results-assessment form (part of the reporting requirement) 6.
• and the final evaluation report (25-30 pages without annexes), the final executive summary and the results-assessment form (part of the reporting requirement)

All reports need to be written in English.

The executive summary should summarize key findings and recommendations (three to five pages) and needs to be submitted as part of the final draft report. The executive summary will include but not limited to overview of evaluation, evaluation objectives and intended audience, evaluation methodology, most important findings and conclusions and main recommendations.

The findings and recommendations of the draft final report and final report have to be structured according to the evaluation questions. An outline of the report’s structure needs to be agreed upon during the inception phase.

The quality of the reports will be judged according to the following criteria:

• Is the results-matrix format part of the report?
• Does the report contain a comprehensive and clear executive summary?
• Were the Terms of Reference fulfilled and is this reflected in the report?
• Is the report structured according to the OECD/DAC criteria?
• Are all evaluation questions answered?
• Are the methods and processes of the evaluation sufficiently documented in the evaluation report?
• Does the report describe and assess the intervention logic (e.g. logframe, program theory) and present/analyze a theory of change and its underlying assumptions?
• Are cross-cutting issues analyzed in the report?
• Are the conclusions and recommendations based on findings and are they clearly stated in the report?
• Does the report clearly differentiate between conclusions, lessons learnt and recommendations?
• Are the recommendations realistic and is it clearly expressed to whom the recommendations are addressed to?
• Were the most significant stakeholders involved consulted?
• Does the report present the information contained in a presentable and clearly arranged form?
• Is the report free from spelling mistakes and unclear linguistic formulations?
• Can the report be distributed in the delivered form?

1. Co-ordination/Responsibility

Mr. Thinley Choedhen will be the contact person for this evaluation.

Contact details: 00975- 77734077, tchoedhen@gnhc.gov.bt

Recommended Presentation of Proposal

Applicants are requested to apply online through procurement.bt@undp.org by latest by 20th May 2019.  The consultancy team are invited to submit applications along with a cover page and the technical proposal with their CV for these positions. The team leader, International Consultant is required to collaborate with the Consultancy team (energy expert and national team member) through a signed letter of association to form the consultancy team. If selected for the assignment, the team Leader of the consultancy team will be fully accountable for the execution of the work including deliverables.

• Duly accomplished Letter of Confirmation of Interest and Availability;
• Personal CV or P11, indicating all past experience from similar projects, as well as the contact details (email and telephone number) of the Candidate and at least three professional references;
• Brief description of why the individual considers him/herself as the most suitable for the assignment, and a methodology on how they will approach and complete the assignment.
• Financial Proposal that indicates the all-inclusive fixed total contract price, supported by a breakdown of costs. The consultant shall submit the price offer indicating a lump sum all-inclusive cost for the assignment with the Technical Proposal.

1. Selection Criteria

The following criteria shall serve as a basis for evaluating the offers:

Combined Scoring method:

• Technical evaluation comprising of 70%, (weightage) and
• Financial evaluation of 30%

1. Documentation

The relevant project documents will be shared with the evaluator for their reference. Following are the list of relevant documents:

• Project Document 13 April 2017
• Annual progress report as of 31.12.2017; as of 30.06.2018
• Minutes of first PSC 4 July 2018, 1st Project Management Meeting 05.02.2018, 2nd Project Management Meeting 12.02. 2019
• Inception Report (undated) of the Expert on Energy Efficiency in Buildings and Construction Materials - United Consultancy, Changangkha, Thimphu, Bhutan, and ZEPHIR - Zero Energy and Passivhaus Institute for Research
• Inception Report 2 March 2018 on Renewable Energy by World Institute of Sustainable Energy and APECS Consultancy
• Training Report 13 September 2018 on Design Aspect of Solar Water Heating Systems
• Workshop Report (undated) on Policy Consultation and Technical Workshop on SWHS.
• 11th FYP (2013 – 2018), Volume I, II https://www.gnhc.gov.bt/en/?page_id=15
• Final Report on 11th FYP (2013 – 2018) July 2018 https://www.gnhc.gov.bt/en/?page_id=15
• Bhutan’s Voluntary National Review Report 2018 on the Implementation of the SDGs July 2018 https://www.gnhc.gov.bt/en/wp-content/uploads/2018/07/VNR_Bhutan_July2018.pdf
• National Energy Efficiency & Conservation Policy (Final) https://www.gnhc.gov.bt/en/wp-content/uploads/2017/05/EEC-Final-Draft-Policy-2017-Final-1.pdf
• Alternative Renewable Energy Policy 2013

1. Annexes:

•  Annex I: Results-Assessment Form, to be filled in by the evaluation team
• Annex II: Audit Trail

ANNEX I:

Annex 1: Results-Assessment Form for End-Term Project Evaluation

This form has to be filled in electronically by the evaluator. No evaluation report will be accepted without this form. The form has to be included at the beginning of the evaluation report.