Background

1.       Background and Context

Disaster management project that was implemented in several phases through UNDP support was a key in developing institutional capacities for disaster risk management in Mongolia. Major accomplishments of the Phase I include the formulation of the Law on Disaster Protection, its approval, and establishment of the National Emergency Management Agency (NEMA) by merging three separate organizations: Civil Defence, the State Reserve, and the State Fire Fighting Department. The Phases II and III supported the implementation of the Law on Disaster Protection through the provision of capacity building opportunities to NEMA and its local branches. A significant policy level outcome was a completion and a formal adoption of a “National Policy on Disaster Protection” and a “National Programme on Strengthening Disaster Protection Capacity”. At the local level, Community-based Disaster Management system was piloted and the community groups made notable progresses in making their livelihoods more resilient to potential disasters through a series of livelihood support activities.

The Phase IV of disaster management project or the project “Strengthening local level capacities for disaster risk reduction, management and coordination in Mongolia” works towards achievement of the Country Programme Action Plan (CPAP) for 2012-2016 that developed by the Government of Mongolia and the UNDP, contributing to the country programme output "National climate and disaster risk management capacities improved in coordination, communication and networking".

Overall goal of the project is to facilitate a decentralized disaster management through sustainable prevention, response and coordination mechanisms and thus reduce vulnerabilities of urban and rural poor. The objective of the project is to enhance disaster management capacities by clarifying roles and responsibilities, formalizing local-level disaster management mechanisms and applying tailored approaches for disaster prevention, preparedness and response in urban and rural settings.

Three inter-related project outputs that support the achievement of the objective:

Output 1: Policy and regulatory frameworks enable clearer roles and responsibilities for improved disaster risk reduction and management;

Output 2: Local-level disaster management mechanisms have procedures and competencies tailored for urban and rural vulnerabilities;

Output 3: Feasible local-level mechanisms for disaster risk reduction and response further replicated.

The project aims to address a lack of clear institutional arrangements for local disaster risk management and replicate successful models on community-level disaster prevention, preparedness and response generated through earlier phases taking into consideration the specific recommendations of the terminal evaluation of UNDP’s disaster management programme Phase III.

An independent terminal evaluation (TE) for the project is planned in the project document to be taken place three months prior to the terminal Project Board meeting. The TE will cover the whole duration of the project from its starting date in August 2013 to the estimated completion date in December 2016. 

Duties and Responsibilities

2.      Evaluation Purpose

The purpose of the TE is assessment of the achievements and effectiveness of the project by looking at potential impacts and sustainability of outcomes and outputs, including the contribution to enabling DRR policy and regulatory framework, local level capacity development and public awareness and education. The objectives of the evaluation are to:

• Assess the outcomes and outputs achieved through the project interventions;
• Evaluate the effectiveness of project activities in contribution to key objectives of the project;
• Evaluate the impact and sustainability of project activities on the target communities;
• Evaluate the effectiveness of project management and interventions;
• Provide recommendation based on the evaluation findings, particularly on Exit Strategy and sustainability measures.

The TE has an additional purpose of drawing lessons, identifying good practices established as a result of the project and developing recommendations for both of UNDP and the Government of Mongolia that may help for improving the selection, enhancing the design and implementation of similar future projects and activities in the country upon project completion.

3.       Scope of the TE

The project evaluation is to be undertaken in accordance with UN evaluation norms and policies, including Standards for Evaluations in the UN System and UNDP Handbook on Planning, Monitoring and Evaluation for Development Results. The TE must provide evidence based information that is credible, reliable and useful. The TE team will review all relevant sources of information including the Project Document, Annual Project Reports, national strategic and legal documents and any other materials that the team considers useful for this evidence-based evaluation.

The TE team is expected to follow a collaborative and participatory approach ensuring close engagement with the Project Team, government counterparts, the UNDP Country Office and other key stakeholders.

Engagement of stakeholders is vital to a successful TE. It should include interviews with stakeholders, including but not limited to the project team, NEMA, provincial Emergency Management Departments, local government and, etc. The TE team is expected to conduct field missions to target areas. 

The TE will assess the project according to standard evaluation criteria of relevance, effectiveness, efficiency, sustainability and impact using the suggested evaluation questions (to be confirmed in consultation with the team), as elaborated below:

Relevance

• How does the project address country priorities? Was the project concept in line with the national sector development priorities and plans of the country? What is the contribution of the project towards the achievement of the State Policy on Disaster Protection, the National Programme on Strengthening Disaster Protection Capacity and CPAP goals/outputs?
• To what extent the tools/instruments/inputs/strategies applied by the project in supporting effective Government policy implementation and in promoting community based DRR action and the project target beneficiaries are relevant?
• To what extent relevant gender issues were raised in the project design?

Effectiveness

• Do strategies, interventions and practices ensure timely and effective implementation of the project?
• To what extent are the outcomes and outputs achieved?
• What factors contributed to achieving or not achieving intended results and outputs?
• How effective was the project’s Monitoring and Evaluation mechanism?

Efficiency

• Has the project utilized the funding as per the project document and agreed work plan to achieve the projected targets?
• How economically resources or inputs are converted to results?
• What factors and constraints have affected project implementation including technical, managerial, organizational, institutional and socio-economic policy issues in addition to other external factors unforeseen during the project design?
• Was the overall project management as effective and efficient as outlined in the Project Document?
• How was the quality of the execution/implementation of an Implementing Partner or the NEMA?

Sustainability and Impact

• Are there preliminary indications that the project results are likely to be sustainable beyond the project’s lifetime (both at the community and government level)? Assess the degree of sustainability. What are the dimensions of sustainability: economic/financial; social/organizational; technological; environmental?
•  Is there any emerging impact on communities for both men and women? To what extent the impact is identified?
• Are communities likely to initiate other disaster risk reduction initiatives in the vulnerable community?
• What relationships/engagements have been formed through project implementation to date that are likely to continue beyond the life of the project?
• Is there an exit strategy and has it been well communicated all around?

The following cross-cutting issues, to which project donor or Luxembourg Development Cooperation attaches a special importance should be considered in the TE:

• Gender Equality,
• Governance for Development,
• Environment and Climate Change.

Lessons learnt/ Conclusions

• Are there significant lessons or conclusions which can be drawn from the project in terms of effectiveness, efficiency, sustainability and networking? 
• Are proper means of communication established to report on the project progress and intended impact to the public? (Did the project implement appropriate outreach and public awareness interventions/campaigns?)

Recommendations:

• What are the potential recommendations that could complement Exit Strategies of the project and follow ups to further strengthen the DRR management and implementation of the national policy on DRR?

4.       Expected Outputs and Deliverables

The evaluation shall report on the findings focusing on above listed criteria.  The main products from the TE are:

• Inception report;
• Presentation of preliminary findings to CO and project management;
• A draft report;
• Responses to the comments on the draft report briefly explaining how the team addressed the comments;
• A final evaluation report. It will be an independent and comprehensive document with annexes as necessary. However, the main report (English and Mongolian) should not exceed 50 pages. 3 hard copies of the final, bound report to UNDP for distribution shall be submitted and an electronic copy (MS Word) of the report included.
• Presentation of the final findings to the IP and stakeholders.

The minimum requirements for the content of the final version of the TE report are:

Executive summary

• Brief description of project
• Context and purpose of the evaluation
• Main conclusions, recommendations and lessons learned

Introduction

• Purpose of the evaluation
• Methodology and scope of the evaluation
• Structure of the evaluation

The project and its development context

• Project start and its duration
• Problems that the project seek to address
• Immediate and development objectives of the project
• Main stakeholders
• Results expected

Findings and ratings

• Project implementation:
• Results:
  • Output I
  • Output II
  • Output III
• Lessons learnt
• Conclusions and recommendation

In addition to a descriptive assessment, all criteria marked with (R) should be rated using the following divisions: Highly Satisfactory, Satisfactory, Marginally Satisfactory, and Unsatisfactory.

1. Project Implementation

Implementation Approach (R). This should include assessments of the following aspects:  1). The use of the logical framework as a management tool during implementation; 2). The general operational relationships between the institutions involved and others and how these relationships have contributed to effective implementation and achievement of project objectives; 3). Technical capacities associated with the project and their role in project development, management and achievements.

Monitoring and evaluation (R). Including an assessment as to whether there has been adequate periodic oversight of activities during implementation; to establish the extent to which inputs, work schedules, other required actions and outputs proceed according to plan; whether action has been taken on the results of this monitoring oversight.

Stakeholder participation (R). This should include assessments of the mechanisms for information dissemination in project implementation and the extent of stakeholder participation in management, emphasizing on 1). the production and dissemination of information generated by the project; 2) local beneficiary participation in project implementation and decision making and an analysis of the strengths and weaknesses of the approach adopted by the project; 3) the establishment of partnerships and collaborative relationships developed by the project with local, national and international entities and the effects they have had on project implementation; 4) Involvement of governmental institutions in project implementation; the extent of governmental support to the project.

Financial Planning: An assessment of the actual project cost by objectives and outputs; the cost-effectiveness of achievements; financial management and co-financing.

Sustainability. Extent to which the benefits of the project will continue, within or outside the project domain, after it has come to an end. Relevant factors that contributed to the sustainability including development of a sustainability strategy, establishment of financial and economic instruments and mechanisms, mainstreaming project objectives into the economy or community production activities.

Execution and implementation modalities. This should consider the effectiveness of the UNDP counterpart and Project Implementation Unit (PIU) participation in selection, recruitment, assignment of experts, consultants and national counterpart staff members and in the definition of tasks and responsibilities; enactment of necessary legislation and budgetary provisions; and extent to which these may have affected implementation and sustainability of the Project; quality and timeliness of inputs by UNDP and GoM and other parties responsible for providing inputs to the project.

         2. Results

Attainment of Outcomes/outputs and Achievement of objectives (R): Including a description and rating of the extent to which the project's objectives were achieved using Highly Satisfactory, Satisfactory, Marginally Satisfactory, and Unsatisfactory ratings. If the project did not establish a baseline (initial conditions), the evaluators should seek to determine it through the use of special methodologies so that achievements, results and impacts can be properly established.

This section should also include reviews of the sustainability output by output including an appreciation of the extent to which benefits continue, within or outside the project domain after Government of Luxemburg/UNDP’s assistance/external assistance has come to an end and contribution to upgrading skills of the national staff.

       3. Lessons learned

This should highlight the best and worst practices in addressing issues and in achieving the results during the project implementation. What are the lessons learnt in terms of the design, approach, partnership and intervention strategies, formulation of policies and legislation, and management structure. 

       4. Recommendations

This should focus on:

• Actions to follow up or reinforce benefits from the project;
• Proposals for future directions underlining the further needs.

1. Annexes
2. Evaluation TOR;
3. Field trip Itinerary;
4. List of persons interviewed;
5. Summary of field visits;
6. List of documents reviewed;
7. Questionnaire and focus group questions used and summary of results;
8. Comments by stakeholders (only in case of discrepancies with evaluation findings and conclusions).

5.  Institutional Arrangements

A team of one international independent consultant and a national consultant hired by UNDP Mongolia will conduct the TE. The Team Leader (international consultant) shall have experience and exposure to evaluations of the projects in developing countries and Asia Pacific region.  The TE team will report to the Implementing Partner and UNDP on the status of the work.

The PIU will provide any necessary logistical support. The project staff will assemble the suggested documents and prepare for the field trip. The evaluation team shall use the project office space. The consultants are expected to bring their own computers/laptops for the written work.

The evaluation consultants should not have participated in the project preparation, formulation, and/or implementation (including the writing of the Project Document) and should not have a conflict of interest with project’s related activities. The national consultant will provide fulltime assistance (logistic, translation, etc.) to the international consultant for all the TE duration in liaison with the PIU and UNDP.

Evaluation will be conducted in accordance with the principles in the UNEG “Ethical Guidelines for Evaluation” to safeguard the rights and confidentiality of information providers including the measures to ensure compliance with legal codes governing areas such as provisions to collect and report data, particularly permissions needed to interview or obtain information about children and young people; provisions to store and maintain security of collected information; and protocols to ensure anonymity and confidentiality.

International Consultant/ Team Leader

The Team Leader will have overall responsibility for the work and operation of the evaluation team, including the coordination of inputs from national team member. The Team Leader is responsible and accountable for the production of the agreed deliverables.

The Team Leader is responsible for the following:

• Desk research of existing management plans, survey/research/reports and databases;
• Conduct fieldwork and interview stakeholders, and communities to generate authentic information and opinions;
• Write and compile the information and reports as needed;
• Make a presentation of key findings highlighting achievements, constraints, and make practical recommendations to decision makers and stakeholders;
• Draft and finalize the Evaluation Report.

National Consultant

The national consultant will assist and collaborate with the Team Leader in all the tasks mentioned above including fieldwork, logistic arrangement in cooperation with PIU, desk-based translation, and assist with interpretation in the field as well as the translation of the final report. The national consultant will be mobilized several days before the Team Leader in an effort to collect data related to the project beforehand. Specific tasks as following:

• Desk review of project materials and databases;
• Fieldwork together with international consultant. Carry out stakeholders’ interview as requested by the team leader and do interpretation work (if necessary);
• Write brief notes, or certain parts of the evaluation report as agreed with the team leader;
• Provide inputs either by written or verbally to the presentation, highlighting key findings, achievements and constraints;
• Contribute to draft and final Evaluation Reports;
• Translate the draft and final reports into Mongolian.

6. Duration of the TE

The total duration of the TE will be 27 working days over approximately 6 weeks starting on 19 August 2016. The tentative TE time-frame is as follows:

1. July 25, 2016:                      Application closes
2. August 19, 2016:                 Selection of TE Team
3. August 19, 2016:                 Prep the TE Team (handover of project documents)
4. August 26 (3 days):             Document review and preparing TE Inception Report
5. September 2 (2 days):         Finalization and Validation of TE Inception Report- latest  Start of TE mission
6. September 5  (10 days):      TE mission: stakeholder meetings, interviews, field visits
7. September 14:                     Mission wrap-up meeting & presentation of initial findings-  earliest end of TE mission
8. September 26 (7 days):        Preparing and submitting draft report
9. October 05 (5 days):            Finalization of TE report, submission and presentation to the IP and the stakeholders.

PS: 1. The date start of contract is August 19, 2016

2. Detailed mission schedule to be developed by the Project Unit and the TE team.

3. The project shall be arranging the transportation during mission in UB as well as field trips.

3. For travel, Consultants are required to comply with the UN security directives set forth under https://dss.un.org/dssweb

7. Qualifications of the Successful Individual Contractor

The selection of the team will be aimed at maximizing the overall “team” qualities in the following areas:

The qualifications and skills required for International Consultant (Team Leader)

• A Master’s degree in development study, disaster risk management, environmental science or other closely related field (10%).
• Recent experience with result-based management evaluation methodologies (25%);
• Experience working/conducting evaluations in developing countries (20%);
• Work experience in relevant technical area of disaster risk management for at least 5 years (20%);
• Demonstrated understanding of issues related to gender and disaster risk/climate change adaptation (5%);
• Excellent communication skills (5%);
• Demonstrable analytical skills (5%);
• Project evaluation/review experiences within United Nations system will be considered an asset (10%).

8. Scope of Price Proposal and Schedule of Payments

The UNDP standard method of payment is the output-based all-inclusive lump-sum scheme.

The payment will be made in three installments upon satisfactory completion of the following deliverables:

1st installment – 10% upon the submission of inception report;

2nd installment - 30% upon the submission of draft report to the PIU and UNDP;

3rd installment –60% upon the submission of final report (English and Mongolian) and approval of report.

* Condition for payment release: within thirty (30) days from the date of UNDP’s written acceptance of the quality of the outputs.                                  

9. Application and Recommended Presentation of Offer

Qualified and interested candidates are requested to apply on-line using this site www.jobs.undp.org

Due to limitation in uploading several documents, applicants are advised to compile all documents into a single attachment and upload the attachment while applying on line.

The following documents will be requested for the positions of international and national consultants:

1. Technical proposal:

• Expression of interest;
• Personal CV or P11, indicating all past experience from similar projects, as well as the contact details (email and telephone number) of the Candidate;
• At least three (3) professional references;
• Brief description of why the individual considers him/herself as the most suitable for the assignment, and a methodology, if applicable, on how they will approach and complete the assignment.

        2. Financial proposal:

Financial Proposal that indicates the all-inclusive fixed total contract price, supported by a breakdown of costs, as per template provided (downloadable from UNDP Mongolia website).  The term “all inclusive” implies all cost (professional fees, travel costs, living allowances etc.). If an Offeror is employed by an organization/company/institution, and he/she expects his/her employer to charge a management fee in the process of releasing him/her to UNDP under Reimbursable Loan Agreement (RLA), the Offeror must indicate at this point, and ensure that all such costs are duly incorporated in the financial proposal submitted to UNDP. 

While preparing your proposal, kindly note that the standard for all travel authorized by UNDP for individual subscribers is economy class and the UN’s Daily Subsistence Allowance (DSA) rates USD197 for Ulaanbaatar and USD72 for the country side in the project target sites, which should provide indication of the cost of living in a duty station/destination. Individuals on this contract are not UN staff and are therefore not entitled to DSAs.  All living allowances required to perform the demands of the ToR must be incorporated in the financial proposal.

10. Criteria for Selection of the Best Offer

The following method will be used for evaluating offers:

• Combined Scoring method – where the qualifications and methodology will be weighted a max. of 70%, and combined with the price offer which will be weighted a max of 30%.

11. Annexes to the TOR

ANNEX 1.

List of documents to be reviewed:

• UNEG evaluation policy and evaluation standards;
• Project Document;
• Annual Work Plans and Reports;
• Mission reports;
• Project Board Meeting minutes;
• Financial and audit report;
• The draft of the Reformulated Law on Disaster Protection of Mongolia;
• The National Programme on Disaster Risk Reduction with Community Participation;
• The Disaster Preparedness Plan of Mongolia;
• The Procedure for Disaster Risk Reduction Volunteers;
• The Model of Soum and Khoroo Disaster Preparedness Plan;
• The Soum Disaster Risk Reduction Programmes;
• The methodology of Disaster Risk Assessment;
• Disaster Protection Curriculums;
• Disaster Protection Manuals and Handbooks;
• Related national laws, sector strategies and policies;
• Others (meeting minutes, correspondence and TORs as needed)

List of key persons to be interviewed during the evaluation field mission:

Ulaanbaatar:

• UNDP Country Office;
• Selected Project Board Members;
• National Project Director and Alternate
• Director of Policy and Cooperation Department;
• Members of the Project Technical Committee;
• Directors of Chingeltei and Songinokhairkhan Emergency Management Departments;
• National Agency of Meteorology and Environmental Monitoring
• Khoroo Governors and officials;
• Self-Help Group members.

Aimag level:

• Aimag governors;
• Directors of Aimag Emergency Management Departments;

Project sites

• Soum governors;
• Local Partnership and Cooperation Council members;
• Project local coordinators;
• Members of forest cooperatives and herder groups;
• Officials responsible for community training classrooms.

Competencies

• Excellent communication skills;
• Demonstrable analytical skills;

Required Skills and Experience

Education:

• A Master’s degree in development study, disaster risk management, environmental science or other closely related field;

Experience:

• Recent experience with result-based management evaluation methodologies;
• Experience working/conducting evaluations in developing countries;
• Work experience in relevant technical area of disaster risk management for at least 5 years;
• Demonstrated understanding of issues related to gender and disaster risk/climate change adaptation;
• Project evaluation/review experiences within United Nations system will be considered as an asset.