Background

The second phase of the USAID funded project on “Enhancing Institutional and Community resilience to disasters and climate change” will be ending in December 2020. As per the UNDP evaluation guidance, conducting a “Terminal Evaluation” during project closure is mandatory. The evaluation must aim to address the extent to which the project has been able to develop resilient cities through risk reduction in the context of disaster and climate change. The evaluation must provide evidence-based information that is credible, reliable and useful. The evaluator is expected to follow a participatory and consultative approach ensuring close engagement with government counterparts The results of the terminal evaluation will be presented to the Implementing partner (Ministry of Home Affairs, Government of India) and will be used to highlight success stories and lesson learning for future endeavours. 

Duties and Responsibilities

Objectives: The objectives of the terminal evaluation is to assess the achievement of project results, and to draw lessons that can both improve the sustainability of benefits from this project, and aid in the overall enhancement of UNDP programming. Accordingly, proposed evaluation of the project will undertake:

• Outcome analysis - what and how much progress has been made towards the achievement of the outcome (including contributing factors and constraints);
• Output analysis - the relevance of and progress made in terms of the UNDP outputs (including analysis of both project and non-project activities);
• The evaluation report must include a chapter providing a set of conclusions, recommendations and lessons 

Scope: Project intervention areas include six cities- Cuttack, Navi Mumbai, Shimla, Shillong, Vishakapatnam and Vijayawada.

Review Criteria and key guiding questions

An assessment of project performance will be carried out, based against expectations set out in the Project Logical Framework/Results Framework, which provides performance and impact indicators for project implementation along with their corresponding means of verification. The evaluation will cover the criteria of: relevance, effectiveness, efficiency, sustainability and impact. Ratings must be provided on the following performance criteria.

Methodology

As of 11 March 2020, the World Health Organization (WHO) declared COVID-19 a global pandemic as the new coronavirus rapidly spread to all regions of the world. If it is not possible to travel to or within the country for the evaluation then the evaluation team is expected to develop a methodology that takes this into account the conduct of the evaluation virtually and remotely, including the use of remote interview methods and extended desk reviews, data analysis, surveys and evaluation questionnaires. This should be detailed in the Inception report and agreed with the Evaluation Manager. If all or part of the evaluation is to be carried out virtually then consideration should be taken for stakeholder availability, ability or willingness to be interviewed remotely. In addition, their accessibility to the internet/ computer may be an issue as many government and national counterparts may be working from home. These limitations must be reflected in the evaluation report. If a data collection/field mission is not possible then remote interviews may be undertaken through telephone or online (skype, zoom etc.) 

The evaluation should employ a combination of both qualitative and quantitative evaluation methods and instruments.

• Document review of all relevant documentation. This would include a review of inter alia (will be provided to selected candidate on Day 1 of assignment)

1. Project document (contribution agreement).
2. Programme and project quality assurance reports.
3. Consolidated quarterly and annual reports. (APRs/PIRs)
4. Project budget
5. Mid-term review / progress reports
6. Results-oriented monitoring report.
7. Highlights of project board meetings.   

• Semi-structured interviews with key stakeholders including key government counterparts, donor community members, representatives of key civil society organizations and implementing partners:

1. Development of evaluation questions around relevance, effectiveness, efficiency and sustainability and designed for different stakeholders to be interviewed.
2. Key informant and focus group discussions with men and women, beneficiaries and stakeholders.
3. All interviews should be undertaken in full confidence and anonymity. The final evaluation report should not assign specific comments to individuals.

• Surveys and questionnaires including participants in development programmes, and/or surveys and questionnaires involving other stakeholders at strategic and programmatic levels.
• The evaluator is expected to follow a participatory and consultative approach that ensures close engagement with the evaluation managers, implementing partners and direct beneficiaries.
• Other methods such as outcome mapping, observational visits, group discussions, etc may be used.
• Data review and analysis of monitoring and other data sources and methods. 

1. Ensure maximum validity, reliability of data (quality) and promote use; the evaluation team will ensure triangulation of the various data sources.

It is preferable that the  interviews/questionnaires with the Ministry of Home Affairs will need to take place on a face to face basis in Delhi. Interviews will also be held with the following organizations and individuals at a minimum: 

1. Joint Secretary, Disaster Management, Ministry of Home Affairs (face to face meeting)
2. Programme Management specialist – Disaster management, USAID
3. Chief, Climate Change, Resilience and Energy, UNDP
4. Municipal Commissioners/representatives of Shimla, Navi Mumbai, Vishakhapatnam, Vijayawada, Cuttack and Shillong. 

1- Evaluation products (deliverables)- refer to Annex for templates. 

• Evaluation inception report: The inception report should be carried out following and based on preliminary discussions with UNDP after the desk review, and should be produced before the evaluation starts (before any formal evaluation interviews, survey distribution or field visits)
• Evaluation debriefings. Immediately following an evaluation, UNDP may ask for a preliminary debriefing and findings.
• Draft evaluation report: The programme unit and key stakeholders in the evaluation should review the draft evaluation report and provide an amalgamated set of comments to the evaluator within an agreed period of time, addressing the content required  and quality criteria as outlined in these guidelines.
• Evaluation report audit trail. Comments and changes by the evaluator in response to the draft report should be retained by the evaluator to show how they have addressed comments.
• Final evaluation report.
• Presentations to the Ministry of Home Affairs and/or the evaluation reference group  

Competencies

Functional:

The evaluation team will comprise of 2 team members (all national) as follows:

• One Lead Consultant  who will be the overall lead to coordinate and manage the Terminal evaluation in accordance with the objective, scope, quality and timely submission of the deliverables; provide strategic guidance; consolidate, review, quality assure, and finalize the reports and deliverables; lead discussions, presentations and sharing of key findings to the government and other stakeholders; and lead briefing meetings and presentations to UNDP India team. 
• The Lead Consultant should have Master’s degree (Ph.d. would be an advantage) in environment, disaster management or climate change field with at least 10 years of broader development experience. Previous experience with results-based monitoring and evaluation methodologies.  Experience on handling projects around disaster risk management and climate change adaptation / mitigation
• One Consultant to support the Lead Consultant:  “Minimum 5  years of relevant professional experience with post graduate degree in engineering/ environment/ management or related filed domain. Previous experience with results-based monitoring and evaluation methodologies.

Key technical and core competencies:

Knowledge of the political, social, cultural, and economic context of the Country; Ability to lead informed and strategic discussions with government counterparts and stakeholders;

• Ability to undertake gender sensitive review and analysis and demonstrated understanding of issues related to gender-mainstreaming and other cross cutting development issues;
• Knowledge of qualitative and quantitative evaluation methods; Proven experience in data analysis as well as report writing; ensuring the triangulation of the findings, obtaining strong evidence for the analysis of information from multiple sources;
• Strong analytical skills (quantitative and qualitative) and strong ability to communicate and summarize this analysis in writing;
• Experience of conducting stakeholder interviews and collecting data remotely;
• Ensuring comments and feedback of the stakeholders is sufficiently addressed in the draft and final reports.
• Deliver the agreed products to the right standard and quality;
• Excellent inter-personal, teamwork and communication skills;
• Flexibility in remote working arrangements and experience of conducting stakeholder interviews and collecting data remotely;
• Openness to change and ability to receive and integrate feedback;
• Excellent report writing, presentation and editing skills in English;
• Keep to the standards and ethical principles in line with UNEG norms and standards and Ethical Guidelines. 

The evaluators selected should not have participated in the project preparation and/or implementation and should not have conflict of interest with project related activities.

Payment Schedule:

Required Skills and Experience

Evaluation Method & Criteria:

The consultant would be selected based on the following criteria:

Technical: 70 points

Financial: 30 points

Criteria for Technical Evaluation would be as under:

*  Educational Qualification, relevant development and work experience – 50 marks

*  Technical Proposal – 20 marks

Technically qualified consultants will be requested to submit their daily fee rate i.e. consultants who score more than 70% i.e. 49 marks with respect to the above-mentioned evaluation criteria. Consultant should not specify their consultancy fee on their CV or with the submission. The CV will not be evaluated further in case the consultant submits the same.

The Consultant is required to submit the following documents, in a single combined PDF file, as the system has provision for uploading only one attachment:

-        Indicate upfront the position for which you are applying (i.e. Lead Consultant (or) Consultant)

-        Personal CV with relevant experience to the TOR; and at least 3 professional references 

-        Short technical proposal (max 2-pages) including methodology, approach & assessment criteria, process followed, data collection and analytical tools.

-        No Financials (Daily Fee) to be submitted at this stage.

Evaluation  Ethics:

This evaluation will be conducted in accordance with the principles outlined in the UNEG ‘Ethical Guidelines for Evaluation’. The consultant must safeguard the rights and confidentiality of information providers, interviewees and stakeholders through measures to ensure compliance with legal and other relevant codes governing collection of data and reporting on data. The consultant must also ensure security of collected information before and after the evaluation and protocols to ensure anonymity and confidentiality of sources of information where that is expected. The information knowledge and data gathered in the evaluation process must also be solely used for the evaluation and not for other uses with the express authorization of UNDP and partners 

The Consultants will be held to the highest ethical standards and are required to sign a Code of Conduct upon acceptance of the assignment. 

Outcome evaluation sample questions

Relevance

•             To what extent is the initiative in line with the UNDP mandate, national priorities of Disaster Management and the requirements of the Urban Local bodies.?

•             To what extent is UNDP support relevant to the achievement of the SDGs in the country?

•             To what extent is UNDP engagement a reflection of strategic considerations, including the role of UNDP in a particular development context and its comparative advantage?

•             To what extent was the method of delivery selected by UNDP appropriate to the development context?

•             To what extent was the theory of change presented in the outcome model a relevant and appropriate vision on which to base the initiatives?

Effectiveness

•             To what extent has progress been made towards outcome achievement? What has been the UNDP contribution to the observed change?

•             What have been the key results and changes attained? How has delivery of country programme outputs led to outcome-level progress?

•             Have there been any unexpected outcome-level results achieved beyond the planned outcome?

•             To what extent has UNDP improved the capacities of national implementing partners to advocate on environmental issues, including climate change issues and disaster risk reduction?

•             To what extent has UNDP partnered with civil society and local communities to promote environmental and disaster risk awareness in the country?

•             To what extent have the results at the outcome and output levels generated results for gender equality and the empowerment of women?

•             To what extent have marginalized groups benefited?

•             To what extent have triangular and South-South cooperation and knowledge management contributed to the results attained?

•             Which programme areas are the most relevant and strategic for UNDP to scale up or consider going forward?

Efficiency

•             To what extent have the programme or project outputs resulted from economic use of resources?

•             To what extent were quality country programme outputs delivered on time?

•             To what extent were partnership modalities conducive to the delivery of country programme outputs?

•             To what extent did monitoring systems provide management with a stream of data that allowed it to learn and adjust implementation accordingly?

•             To what extent did UNDP promote gender equality, the empowerment of women, human rights and human development in the delivery of country programme outputs?

•             To what extent have UNDP practices, policies, processes and decision-making capabilities affected the achievement of the country programme’s outcomes?

•             To what extent did UNDP engage or coordinate with beneficiaries, implementing partners, other United Nations agencies and national counterparts to achieve outcome-level results?

Sustainability

•             To what extent did UNDP establish mechanisms to ensure the sustainability of the country programme outcomes?

•             To what extent do national partners have the institutional capacities, including sustainability strategies, in place to sustain the outcome-level results?

•             To what extent are policy and regulatory frameworks in place that will support the continuation of benefits?

•             To what extent have partners committed to providing continuing support (financial, staff, aspirational, etc.)?

•             To what extent do mechanisms, procedures and policies exist to carry forward the results attained on gender equality, empowerment of women, human rights and human development by primary stakeholders?

•             To what extent do partnerships exist with other national institutions, NGOs, United Nations agencies, the private sector and development partners to sustain the attained results?

Evaluation cross-cutting issues sample questions

Gender equality

• To what extent have gender equality and the empowerment of women been addressed in the design, implementation and monitoring of the project?
• Is the gender marker data assigned to this project representative of reality?
• To what extent has the project promoted positive changes in gender equality and the empowerment of women? Were there any unintended effects? 

Note: Please ensure that all the documents to be uploaded should be combined in a single PDF file before uploading as the system has provision of uploading only one document.

General Conditions for Individual Contract-
http://www.in.undp.org/content/dam/india/docs/procurement/UNDP%20General%20Conditions%20for%20Individual%20Contracts.pdf 

For any clarifications, please write to: manikandan.srinivasan@undp.org 

Please apply online by visiting www.in.undp.org (Careers).