Background
Umbrella Programme for Socio-Economic Development
Mid-Term Evaluation TOR
- Background and context
The Umbrella Programme for Socio-Economic Development Project, launched in May 2019, is designed to support mainstreaming social and economic development in the national policies in Saudi Arabia. The ultimate programme objective is to strengthen evidence-based policy planning and decision-making to support the realization of Saudi Vision 2030 and the Sustainable Development Goals (SDGs).
The review of the first year of the project implementation identified a need for additional programme efforts to enable the project achieves its objectives. As a result, UNDP and MOEP have agreed to revise the Umbrella Programme for Socio-Economic Development Project to increase policy engagement, reinforce capacity development and strengthen monitoring and evaluation. The revised project will enable more coherent and structured support to policy-making, policy planning and assessment. The amendment takes into account the needs for technical support of the various deputyships in MOEP as well as the support needed to strengthen national and sub-national partners’ engagement in socio-economic policy-making. The amended programme framework will cover the following main areas:
• Strengthening MOEP capacity for policy-making, policy analysis and evaluation
• SDGs monitoring and reporting
• Support to regional development planning
• Producing the National Human Development Report
• Assessing the impacts of the COVID-19 on the Saudi economy
Duties and Responsibilities
Basic project information can also be included in table format as follows:
PROJECT/OUTCOME INFORMATION | ||
Project/outcome title | Umbrella Programme for Socio-Economic Development | |
Atlas ID | SAU10- 00113712 | |
Corporate outcome and output | Outcome 1: Improved knowledge-based equitable and sustainable development underpinned by innovation and improved infrastructure. | |
Country | Saudi Arabia | |
Region | RBAS | |
Date project document signed | 9 May, 2019 | |
Project dates | Start | Planned end |
1 May, 2019 | 31 December 2024 | |
Project budget | $ 12,000,000 | |
Project expenditure at the time of evaluation | 3,773,458 | |
Funding source | Government | |
Implementing party[1] | Ministry of Economy and Planning |
- Evaluation purpose, scope and objectives
This is a mandatory mid-term evaluation planned half-way through the life cycle of the project and triggered by a substantial budget revision with an increased budget and new components in terms of outputs of the project. As such, it was deemed crucial to undertake this evaluation to ensure the project is on track to meet its targets and deliver the services expected with the impacts its due to demonstrate.
The project has been ongoing since May 2019 and has, thus far, never been evaluated. Drastic changes have been taking place in the country and the project has had to adapt to the changes over recent years, this included changes in Ministers, Deputy Ministers and Project staff, resulting in changing project directions. To ensure the project has delivered its intended objectives thus far and to provide recommendations for the way forward, it is imperative to conduct a mid-term evaluation and ensure the project delivery is on track. This evaluation will benefit the Ministry of Economy and Planning in their planning for future years to meet Saudi Vision 2030 and highlight the impacts this project has had on the economy sector over the past few years.
This evaluation will measure the performance in the past two years of the project in terms of delivering on:
- Strengthening MOEP capacity for policy-making, policy analysis and evaluation
- SDGs monitoring and reporting
- Support to regional development planning
- Producing the National Human Development Report
- Assessing the impacts of the COVID-19 on the Saudi economy
The evaluation will also address the primary issues of concern to the national partners, namely MEOP and provide recommendations as to the best course of action that needs to be taken to ensure the achievements of intended results.
This evaluation will also provide recommendations as to how the project will, in the future, mainstream gender in development efforts, and, wherever possible, also consider disability issues and the rights-based approach
- Evaluation criteria and key guiding questions
Evaluation questions define the information that the evaluation will generate. To assess the impact of this project in the grand scheme of Outcomes, the following are guiding and crucial, but in no way exclusive, questions the evaluation must answer to provide the information needed in order to make decisions, take actions or increase knowledge.
Questions are grouped according to the four or five OECD-DAC evaluation criteria: (a) relevance; (b) coherence; (c) effectiveness; (d) efficiency; and (e) sustainability.
| |
Relevance/ Coherence
Effectiveness
Efficiency
Sustainability
|
|
Evaluation questions on cross-cutting issues
Human rights
Gender equality All evaluation criteria and evaluation questions applied need to be checked to see if there are any further gender dimensions attached to them, in addition to the stated gender equality questions.
Disability
|
- Methodology
This TOR suggests an overall approach and method for conducting the evaluation, as well as data sources and tools that will likely yield the most reliable and valid answers to the evaluation questions within the limits of resources. However, final decisions about the specific design and methods for the evaluation should emerge from consultations with the programme unit, the evaluator and key stakeholders about what is appropriate and feasible to meet the evaluation purpose and objectives and answer the evaluation questions, given limitations of budget, time and data. This will have to be reflected in the inception report prior to starting the evaluation mission.
Evaluation should employ a combination of qualitative and quantitative evaluation methods and instruments. The evaluator is expected to follow a participatory and consultative approach that ensures close engagement with the evaluation managers, implementing partners and male and female direct beneficiaries. Suggested methodological tools and approaches may include:
- Document review. This would include a review of all relevant documentation, inter alia
- Project document (contribution agreement).
- Theory of change and results framework.
- Programme and project quality assurance reports.
- Annual workplans.
- Activity designs.
- Consolidated quarterly and annual reports.
- Results-oriented monitoring report.
- Highlights of project board meetings.
- Technical/financial monitoring reports.
- Interviews and meetings with key stakeholders (men and women) such as key government counterparts, donor community members, representatives of key civil society organizations, United Nations country team (UNCT) members and implementing partners:
- Semi-structured interviews, based on questions designed for different stakeholders based on evaluation questions around relevance, coherence, effectiveness, efficiency, and sustainability.
- Key informant and focus group discussions with men and women, beneficiaries and stakeholders.
- All interviews with men and women should be undertaken in full confidence and anonymity. The final evaluation report should not assign specific comments to individuals.
- Surveys and questionnaires including male and female participants in development programmes, UNCT members and/or surveys and questionnaires to other stakeholders at strategic and programmatic levels.
- Field visits and on-site validation of key tangible outputs and interventions.
- Other methods such as outcome mapping, observational visits, group discussions, etc.
- Data review and analysis of monitoring and other data sources and methods. To ensure maximum validity, reliability of data (quality) and promote use, the evaluation team will ensure triangulation of the various data sources.
- Gender and human rights lens. All evaluation products need to address gender, disability, and human right issues.
The final methodological approach including interview schedule, field visits and data to be used in the evaluation should be clearly outlined in the inception report and fully discussed and agreed between UNDP, key stakeholders and the evaluators.
- Evaluation products (deliverables)
The TOR should clearly outline the outputs UNDP expects from the evaluation team, with a detailed timeline and schedule for completion of the evaluation products. Where relevant, the TOR should also detail the length of specific products (number of pages). These products include:
- Evaluation inception report (10-15 pages). The inception report should be carried out following and based on preliminary discussions with UNDP after the desk review and should be produced before the evaluation starts (before any formal evaluation interviews, survey distribution or field visits) and prior to the country visit in the case of international evaluators.
- Evaluation debriefings. Immediately following an evaluation, UNDP may ask for a preliminary debriefing and findings.
- Draft evaluation report (within an agreed length). A length of 40 to 60 pages including executive summary is suggested.
- Evaluation report audit trail. The programme unit and key stakeholders in the evaluation should review the draft evaluation report and provide an amalgamated set of comments to the evaluator within an agreed period of time, as outlined in these guidelines. Comments and changes by the evaluator in response to the draft report should be retained by the evaluator to show how they have addressed comments.
- Final evaluation report.
- Presentations to stakeholders and/ or evaluation reference group (if required).
- Evaluation brief and other knowledge products or participation in knowledge-sharing events, if relevant to maximise use.
- Evaluation team composition and required competencies
This section details the specific skills, competencies and characteristics required of the evaluator / individual evaluators in the evaluation team, and the expected structure and composition of the evaluation team, including roles and responsibilities of team members. This may include:
- Required qualifications: Advanced degree in relevant field, a minimum of ten years’ experience in conducting/ managing evaluations, relevant knowledge, and specific country/regional experience.
- Technical competencies: team leadership skills and experience, technical knowledge in UNDP thematic areas, with specifics depending on the focus of the evaluation, data analysis and report writing etc.
- Technical knowledge and experience: Gender expertise/competencies in the evaluation a strong plus. some knowledge and/or experience of disability inclusion. Technical knowledge and experience in other cross-cutting areas such equality, disability issues, rights-based approach, and capacity development.
- Language skills required: Fluent English speaking, writing, and reading skills, Arabic is an asset
The provision of evidence will be expected to support claims of knowledge, skills and experience to the above in terms of:
- resumes,
- work samples,
- references
The Evaluator is to clearly state independence from any organizations that have been involved in designing, executing, or advising any aspect of the intervention that is the subject of the evaluation.[3]
- Evaluation ethics
Explicit statement that evaluations in UNDP will be conducted in accordance with the principles outlined in the UNEG ‘Ethical Guidelines for Evaluation’.[4]
“This evaluation will be conducted in accordance with the principles outlined in the UNEG ‘Ethical Guidelines for Evaluation’. The consultant must safeguard the rights and confidentiality of information providers, interviewees, and stakeholders through measures to ensure compliance with legal and other relevant codes governing collection of data and reporting on data. The consultant must also ensure security of collected information before and after the evaluation and protocols to ensure anonymity and confidentiality of sources of information where that is expected. The information knowledge and data gathered in the evaluation process must also be solely used for the evaluation and not for other uses with the express authorization of UNDP and partners.”
- Implementation arrangements
In order to clarify expectations, eliminate ambiguities, and facilitate an efficient and effective evaluation process, the following is the organization and management structure for the evaluation as well as the definition of the roles, key responsibilities and lines of authority of all parties involved in the evaluation process.
- The evaluation commissioner: Accountable for the quality and approval of final terms of reference (TORs), final evaluation reports and management responses before final submission to the ERC. This is traditionally the head of office, in the case the Resident Representative of UNDP KSA Country Office.
- The evaluation manager: Lead the evaluation process and participate in all of its stages - evaluability assessment, preparation, implementation, management and use. In this case it is the M&E Officer
- The programme manager: Support the establishment of the evaluation reference group with key project partners where needed and participate in calls/meetings on request. Provide inputs/ advice to the evaluation manager on the detail and scope of the TOR for the evaluation and how the findings will be used. Provide the evaluation manager with all required data (e.g. relevant monitoring data) and documentation (reports, minutes, reviews, studies, etc.), contacts/ stakeholder list etc.
- The evaluation partner: Participate in the review of key evaluation deliverables, including the TOR, inception report, and successive versions of the draft evaluation report. Ensure that data and documentation in general, but in particular relating to gender equality and women’s empowerment and other cross-cutting issues, are made available to the evaluation manager
- The independent evaluator: Fulfil the contractual arrangements under the TOR. Develop the evaluation inception report, including an evaluation matrix and a gender responsive methodology, in line with the TOR, UNEG norms and standards and ethical guidelines. Conduct data collection and field visits according to the TOR and inception report. Produce draft reports adhering to UNDP evaluation templates, and brief the evaluation manager, programme/ project managers and stakeholders on the progress and key findings and recommendations. Consider gender equality and women’s empowerment and other cross-cutting issues, check if all and respective evaluation questions are answered, and relevant data, disaggregated by sex, is presented, analysed and interpreted. Finalize the evaluation report, incorporating comments and questions from the feedback/audit trail. Record own feedback in the audit trail
The consultant will be expected to deliver the following:
- Evaluation inception report (10-15 pages). The inception report should be carried out following and based on preliminary discussions with UNDP after the desk review and should be produced before the evaluation starts (before any formal evaluation interviews, survey distribution or field visits) and prior to the country visit in the case of international evaluators.
- Evaluation findings debriefings. Immediately following an evaluation, UNDP may ask for a preliminary debriefing and findings.
- Draft evaluation report (60 pages including executive summary). The programme unit and key stakeholders in the evaluation should review the draft evaluation report and provide a set of comments to the evaluator within an agreed period of time, addressing the content required (as agreed in the TOR and inception report) and quality criteria as outlined in the evaluation guidelines.
- Evaluation report audit trail. Comments and changes by the evaluator in response to the draft report should be retained by the evaluator to show how he/she has addressed comments.
- Final evaluation report:
- Executive summary;
- Introduction, including description of the work conducted;
- Findings and conclusions;
- Recommendations, including, as applicable, a revised work plan to address the pending tasks and eventual corrective action as well as an improved system for measuring the impact of the project in terms of achieved energy savings;
- Annexes providing a brief summary of the documents reviewed and persons interviewed with the description of the key content / conclusions drawn and any other relevant materials.
- Validation workshop for presentations to stakeholders
- Evaluation brief and other knowledge products or participation in knowledge-sharing events, if relevant.
- The consultant should present three hard copies of the report as well as an electronic copy. The draft final report should be submitted no later than three weeks after the end of the on-site mission and the final report within two weeks from receiving the comments of the project management and UNDP on the draft reports
Standard templates that need to be followed are provided in the Annexes section. It is expected that the evaluator will follow the UNDP evaluation guidelines and UNEG quality check list and ensure all the quality criteria are met in the evaluation report.
In line with UNDP’s financial regulations, when determined by the Country Office and/or the consultant that a deliverable or service cannot be satisfactory completed due to impact of COVID-19 and limitations to the evaluation, that deliverable or service will not be paid. Due to the current COVID-19 situation and its implications, a partial payment may be considered if the consultant invested time towards the deliverable but was unable to complete to circumstances beyond his/her/their control.
- Time frame for the evaluation process
Working day allocation and schedule
ACTIVITY | ESTIMATED # OF DAYS | DATE OF COMPLETION | PLACE | RESPONSIBLE PARTY |
Phase One: Desk review and inception report | ||||
Meeting briefing with UNDP (programme managers and project staff as needed) | - | At the time of contract signing
| UNDP or remote | Evaluation manager and commissioner |
Sharing of the relevant documentation with the evaluation team | - | At the time of contract signing
| Via email | Evaluation manager and commissioner |
Desk review, Evaluation design, methodology and updated workplan including the list of stakeholders to be interviewed | 7 days | Within two weeks of contract signing
| Home- based | Evaluation Team |
Submission of the inception report (15 pages maximum) | - | Within two weeks of contract signing
|
| Evaluation team |
Comments and approval of inception report | - | Within one week of submission of the inception report
| UNDP | Evaluation manager |
Phase Two: Data-collection mission | ||||
Consultations and field visits, in-depth interviews, and focus groups | 10 days | Within four weeks of contract signing
| In country
With field visits | UNDP to organize with local project partners, project staff, local authorities, NGOs, etc. |
Debriefing to UNDP and key stakeholders | 1 day |
| In country | Evaluation team |
Phase Three: Evaluation report writing | ||||
Preparation of draft evaluation report (50 pages maximum excluding annexes), executive summary (4-5 pages) | 7 days | Within three weeks of the completion of the field mission
| Home- based | Evaluation team |
Draft report submission | - |
|
| Evaluation team |
Consolidated UNDP and stakeholder comments to the draft report | - | Within two weeks of submission of the draft evaluation report
| UNDP | Evaluation manager and evaluation reference group |
Debriefing with UNDP | 1 day | Within one week of receipt of comments
| Remotely UNDP | UNDP, evaluation reference group, stakeholder, and evaluation team |
Finalization of the evaluation report incorporating additions and comments provided by project staff and UNDP country office | 4 days | Within one week of final debriefing
| Home- based | Evaluation team |
Submission of the final evaluation report to UNDP country office (50 pages maximum excluding executive summary and annexes) | - | Within one week of final debriefing
| Home- based | Evaluation team |
Estimated total days for the evaluation | 30 |
|
|
|
[1] This is the entity that has overall responsibility for implementation of the project (award), effective use of resources and delivery of outputs in the signed project document and workplan.
[2] The twin-track approach combines mainstream programmes and projects that are inclusive of persons with disabilities as well as programmes and projects that are targeted towards persons with disabilities. It is an essential element of any strategy that seeks to mainstream disability inclusion successfully. Also, see chapter 9 of the Technical Notes. Entity Accountability Framework. United Nations Disability and Inclusion Strategy: https://www.un.org/en/disabilitystrategy/resources
[3] For this reason, UNDP staff members based in other country offices, regional centres and headquarters units should not be part of the evaluation team.
[4] UNEG, ‘Ethical Guidelines for Evaluation’, 2020. Access at: http://www.unevaluation.org/document/detail/2866
Competencies
- Technical competencies: team leadership skills and experience, technical knowledge in UNDP thematic areas, with specifics depending on the focus of the evaluation, data analysis and report writing etc.
- Technical knowledge and experience: Gender expertise/competencies in the evaluation a strong plus. some knowledge and/or experience of disability inclusion. Technical knowledge and experience in other cross-cutting areas such equality, disability issues, rights-based approach, and capacity development.
- Language skills required: Fluent English speaking, writing, and reading skills, Arabic is an asset
The provision of evidence will be expected to support claims of knowledge, skills and experience to the above in terms of:
- resumes,
- work samples,
- references
The Evaluator is to clearly state independence from any organizations that have been involved in designing, executing, or advising any aspect of the intervention that is the subject of the evaluation.[1]
[1] For this reason, UNDP staff members based in other country offices, regional centres and headquarters units should not be part of the evaluation team.
Required Skills and Experience
This section details the specific skills, competencies and characteristics required of the evaluator / individual evaluators in the evaluation team, and the expected structure and composition of the evaluation team, including roles and responsibilities of team members. This may include:
- Required qualifications: Advanced degree in relevant field, a minimum of ten years’ experience in conducting/ managing evaluations, relevant knowledge, and specific country/regional experience.
- Application submission process and criteria for selection
As required by the programme unit.
- TOR annexes
Annexes can be used to provide additional detail about evaluation background and requirements to facilitate the work of evaluators. Some examples include:
- Intervention results framework and theory of change. Provides more detailed information on the intervention being evaluated.
- Key stakeholders and partners. A list of key stakeholders and other individuals who should be consulted, together with an indication of their affiliation and relevance for the evaluation and their contact information. This annex can also suggest sites to be visited.
- Documents to be consulted. A list of important documents and web pages that the evaluators should read at the outset of the evaluation and before finalizing the evaluation design and inception report. This should be limited to the critical information that the evaluation team needs. Data sources and documents may include:
- Relevant national strategy documents.
- Strategic and other planning documents (e.g., programme and project documents).
- Monitoring plans and indicators.
- Partnership arrangements (e.g., agreements of cooperation with governments or partners).
- Previous evaluations and assessments.
- UNDP evaluation policy, UNEG norms and standards and other policy documents.
Table 1. Sample evaluation matrix
Relevant evaluation criteria | Key questions | Specific sub-questions | Data sources | Data collection methods/ tools | Indicators/ success standards | Methods for data analysis |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- Schedule of tasks, milestones, and deliverables. Based on the time frame specified in the TOR, the evaluators present the detailed schedule.
- Required format for the evaluation report. The final report must include, but not necessarily be limited to, the elements outlined in the template for evaluation reports (see annex 4 below).
- Dispute and wrongdoing resolution process and contact details (annex 3)
- Pledge of ethical conduct in evaluation. UNDP programme units should request each member of the evaluation team to read carefully, understand and sign the ‘Pledge of Ethical Conduct in Evaluation of the United Nations system’.[1]
[1]http://www.unevaluation.org/document/detail/2866#:~:text=The%20UNEG%20Ethical%20Guidelines%20for%20Evaluation%20were%20first%20published%20in%202008.&text=This%20document%20aims%20to%20support,day%20to%20day%20evaluation%20practice.
Interested individual consultants must submit the following documents/information to demonstrate their qualifications in one single PDF document:
- Technical Methodology/Proposal
- Duly accomplished Confirmation of Interest and Submission of Financial Proposal Template using the template provided by UNDP (Annex II);
- Personal CV or P11, indicating all past experience from similar projects, as well as the contact details (email and telephone number) of the Candidate and at least three (3) professional references.
The interested offeror must read the Individual Consultant (IC) Procurement Notice, which can be viewed at
https://procurement-notices.undp.org/view_notice.cfm?notice_id=81171
for more detailed information about the term of references, instructions to the offeror, and to download the documents to be submitted in the offer online.
UNDP reserves the right to reject any applications that are incomplete.
Please be informed that we don’t accept applications submitted via email.
Interested Offerors are required to submit an application via UNDP Jobsite system as the application screening and evaluation will be done through UNDP Jobsite system. Please note that UNDP Jobsite system allows only one uploading of application documents, so please make sure that you merge all your documents into a single file. Your on¬line application submission will be acknowledged where an email address has been provided. If you do not receive an e¬mail acknowledgement within 24 hours of submission, your application may not have been received. In such cases, please resubmit the application, if necessary. Please combine all your documents into one (1) single PDF document as the system only allows you to upload a maximum of one document.
Any request for clarification/additional information on this procurement notice shall be communicated in writing to UNDP office or send to email mohammed.abbas@undp.org. While the Procurement Unit would endeavour to provide information expeditiously, only requests receiving at least 3 working days prior to the submission deadline will be entertained. Any delay in providing such information will not be considered as a reason for extending the submission deadline. The UNDP's response (including an explanation of the query but without identifying the source of inquiry) will be posted on the Individual Consultant (IC) Procurement Notice page as provided above. Therefore, all prospective Offerors are advised to visit the page regularly to make obtain updates related to this Individual Consultant (IC) Procurement Notice
UNDP is committed to achieving workforce diversity in terms of gender, nationality and culture. Individuals from minority groups, indigenous groups and persons with disabilities are equally encouraged to apply. All applications will be treated with the strictest confidence.
UNDP does not tolerate sexual exploitation and abuse, any kind of harassment, including sexual harassment, and discrimination. All selected candidates will, therefore, undergo rigorous reference and background checks.
Annexes:
Annex-1 – Template Confirmation of Interest and Submission of Financial Proposal.
Annex-2 – Individual Consultant General Terms and Conditions.
Annex 3 : TOR