UNDP Jobs - 67389- OUTCOME Evaluation: Sustainable Development Mains

OUTCOME Evaluation: Sustainable Development Mainstreamed across the Economy

Location :

Riyadh, SAUDI ARABIA

Type of Contract :

Individual Contract

Starting Date :

04-Sep-2016

Application Deadline :

07-Aug-16 (Midnight New York, USA)

Post Level :

International Consultant

Duration of Initial Contract :

35 days

Languages Required :

Arabic English

Expected Duration of Assignment :

35 days

UNDP is committed to achieving workforce diversity in terms of gender, nationality and culture. Individuals from minority groups, indigenous groups and persons with disabilities are equally encouraged to apply. All applications will be treated with the strictest confidence. UNDP does not tolerate sexual exploitation and abuse, any kind of harassment, including sexual harassment, and discrimination. All selected candidates will, therefore, undergo rigorous reference and background checks.

Background

According to the evaluation plan for 2012-2016 of the United Nations Development Programme (UNDP) in Saudi Arabia, an outcome evaluation is to be conducted to assess the impact of programme component of the UNDP’s development assistance: Outcome 3 Sustainable Development Mainstreamed across the Economy.

UNDP in Saudi Arabia would like to evaluate its contribution during 2012-2016 to the achievement of the Outcome and take stock of previous efforts and lessons learned. An outcome evaluation assesses how and why an outcome is or is not being achieved in Saudi Arabia’s context and the role UNDP has played. It is also intended to clarify underlying factors affecting the development situation, identify unintended consequences (positive and negative), generate lessons learned and recommend actions to improve performance in future programming and partnership development. Outcome evaluation also should be able to answer whether UNDP supported the Government of Saudi Arabia in meeting the 9th National Development Plans and the achievement of the Millennium Development Goals.

Brief National Context:

The Kingdom of Saudi Arabia is a high-income country with a per capita GDP of US$ 17,819 and total GDP of US$ 752 billion in 2014. Saudi Arabia has an estimated population of approximately 31 million for 2014 with 21 million Saudis and 10 million expatriates. Almost half the population is below the age of 24. Saudi Arabia’s HDI for 2014 is 0.837 - which places it in the very high human development category - at 39 out of 188 countries and territories.

The Ninth National Development Plan (2009–2014) contributed to continuous economic growth, provision of job opportunities, increased income, expansion of infrastructure projects, establishment of basic industries, expansion of public health, and construction of public facilities. It has resulted in continuous improvement of citizen's standard of living and their quality of life. During this period the real economic growth rate averaged 4 per cent. A number of programmes have been implemented aimed at increasing employment of Saudis. Saudi Arabia has also achieved all the Millennium Development Goals at the national level. However, since 2015, Saudi Arabia’s fiscal performance was marked by the decrease in government revenues mainly driven by lower oil prices and the war in Yemen.

Despite the progress that Saudi Arabia has made over the past years, a number of challenges remain. These include, diversifying the economy; building Saudi human capacities to lead and participate in the new industrial and service sectors, thereby decreasing reliance on foreign workers; more effectively translating national policies into human development gains through improved governance and public administration; developing capacities to ensure that growth proceeds with equity, with respect to issues of human rights, women, youth, the vulnerable and disabled and in different regions; and engaging in global issues of climate change and environment. The adoption and adaptation of the Sustainable Development Goals (SDGs) will provide an opportunity to ensure integrated policy approaches to address the challenges at hand with clearly defined targets for coordinated implementation.

With the high dependence on the petroleum sector, accounting for roughly 80 percent of budget revenues, 45 per cent of GDP, and 90 per cent of export earnings, Saudi Arabia is encouraging the diversification of the economy through the growth of the private sector and increasing employment of Saudi nationals. The government is particularly focused on employing the country’s large youth population, primarily in the private sector. In order to create an active and productive private-sector, raising the skills of Saudi workers through education and training is needed.

Saudi Arabia has made a major achievement with regard to the political and economic empowerment of women who account for 20 per cent of members of Shura Council. Women for the first time voted and stood as candidates in municipal elections in December 2015. The voting age was lowered to 18 to allow for more youth participation. 21 women secured municipal councils seats in different regions of the country. The participation of women in the job market is increasing and new work opportunities within the private sector have been opening. The largest portion of Saudis work in the government sector representing 66% of the total employed Saudis, with males constituting about 53% and females about 13%. Although women participation in higher education is high, the unemployment rate amongst women remains at a high 33.3 per cent. Unemployment is high among youth gradutes from universities reaching 5% among males and as high as 34% among females in 2014. The Gender Inequality Index value is at 0.284 globally ranking at 56. Job matching between graduates and industry needs is a challenge.

Among the sweeping changes for improved strategic direction, policy setting and coordination, HM King Salman created a Council of Economic and Development Affairs (CEDA) comprising of 18 ministries and chaired by the Deputy Crown Prince and Minister of Defense. CEDA has taken the lead on an ambitious and transformative agenda with the release, in April 2016, of the Saudi Vision 2030. This important strategy aims to diversify the economy away from its dependence on oil; establish a huge Public Investment Fund through the sale of 5% of ARAMCO; create job opportunities for youth, with equal opportunities for men and women, primarily in the private sector; focusing on the promotion of small and medium enterprises; privatization of national entities, amongst a number of other actions. A National Transformation Plan (NTP) 2020 is launched specifying targets and projects to be implemented by each line ministry.

The UNDP Country Office will be conducting this outcome evaluation in 2016, which should provide a more evidence-based information on UNDP’s contribution to the development results during the 2012-2016 country programme cycle. To achieve the Outcome on Sustainable Development Mainstreamed across the Economy, the UNDP has focused on enhancing capacities government partners towards national strategy formulation to enhance the economy.

Duties and Responsibilities

The overall objective of the outcome evaluation will be to assess how UNDP’s programme results contributed, together with the assistance of partners, to a change in development conditions, especially in the area of sustainable development. The purpose of the proposed evaluation is to measure UNDP’s contribution to the outcome outlined above with a view to improve on the current and new UNDP programme, providing the most optimal portfolio balance and structure for the next programming cycle (2017-2021). The evaluation should highlight the impact, efficiency, effectiveness and suitability of achievements.

Evaluation scope:

The evaluation will cover UNDP Outcome 3 under the current CPD. This outcome evaluation will assess progress towards the outcome, the factors affecting the outcome, key UNDP contributions to the outcome and assess the partnership strategy. (Please, See to Table 1&2 in the attached ToR)

Outcome status: Determine whether there has been progress made towards the Outcome 3 achievement, and also identify the challenges to attainment of the outcome. Identify innovative approaches and capacities developed through UNDP assistance. Assess the relevance of UNDP outputs to the outcome.

Underlying factors: Analyze the underlying factors beyond UNDP’s control that influenced the outcome. Distinguish the substantive design issues from the key implementation and/or management capacities and issues including the timeliness of outputs, the degree of stakeholders and partners’ involvement in the completion of outputs, and how processes were managed/carried out.

Strategic Positioning of UNDP: Examine the distinctive characteristics and features of UNDP’s inclusive development programme and how it has shaped UNDP's relevance as a current and potential partner. The Country Office (CO) position will be analyzed in terms of communication that goes into articulating UNDP's relevance, or how the CO is positioned to meet partner needs by offering specific, tailored services to these partners, creating value by responding to partners' needs.

Partnership strategy: Ascertain whether UNDP’s partnership building efforts has been appropriate and effective. What were the partnerships formed? What was the role of UNDP? How did the partnership contribute to the achievement of the outcome? What was the level of stakeholders’ participation? Examine the partnership among UN Agencies in the relevant field. This will also aim at validating the appropriateness and relevance of the outcome to the country’s needs and the partnership strategy and hence enhancing development effectiveness and/or decision making on UNDP future role in development.

Lessons learned: Identify lessons learnt and best practices and related innovative ideas and approaches in incubation, and in relation to management and implementation of activities to achieve the related outcome. This will support learning lessons about UNDP’s contribution to the outcomes over the CPD cycle so as to design a better assistance strategy for the new programming cycle.

Outcome evaluation design should clearly spell out the key questions according to the evaluation criteria against which the subject to be evaluated. The questions when answered, will give intended users of the evaluation the information in order to make decisions, take action or add to knowledge. The questions cover the following key areas of evaluation criteria:

Relevance: the extent to which the Outcome activities are suited to the priorities and policies of the country at the time of formulation:

Did the Outcome activities design properly address the issues identified in the country?

Did the Outcome objective remain relevant throughout the implementation phase, where a number of changes took place in the development of Saudi Arabia?

Has UNDP played a role in introducing the Government to the best global practices of public service based on the principles of governance, public sector performance and participatory decision-making?

Have strategies been launched - including a National Spatial Strategy - for regional balance and specialization, and National Rural Development Strategy for connecting communities to services?

To what degree are approaches such as a human rights based approach to programming, gender mainstreaming and results-based management understood and pursued in a coherent fashion?

Efficiency: measurement of the outputs in relation to the inputs.

Have the results been achieved at an acceptable cost, compared with alternative approaches with the same objectives? If so, which types of interventions have proved to be more cost-efficient?

How much time, resources and effort it takes to manage the sustainable development portfolio? Where are the gaps if any?

How did UNDP practices, policies, decisions, constraints and capabilities affect the performance of the sustainable development portfolio?

Effectiveness: the extent to which the Outcome activities attain its objectives.

How many and which of the outputs are on track by 2016?

What progress toward the Outcome delivery has been made by 2016?

What factors have contributed to achieving or not achieving the intended Outcome?

Has UNDP supported the Government to increase accountability, transparency and sensitivity to people needs, especially those who vulnerable?

To what extent has the Cooperation focused on enhancing institutional capacities for results based management and monitoring of NPD results through evidence-based indicator systems at national and local levels including MDG Reports, NHDRs and National and Local Urban Observatory systems?

Has UNDP made impact to improve in transparency and the integrity system of the government?

Sustainability: the benefits of the Programme related activities that are likely to continue after the Programme fund has been exhausted

To what degree have capacities been developed for global partnerships and mechanisms such as MDG review processes and sharing KSA’s development successes with other countries through UN and related conferences and fora?

To what extent have new capacities been developed for expansion of areas of new emphasis like tourism and information technology as well as policies and institutional capacities for shifting to a knowledge economy including the role of New Economic Cities?

How UNDP has contributed to human and institutional capacity building of partners as a guarantee for sustainability beyond UNDP interventions?

Has follow up support after the end of the Outcome activities been discussed and formalized? Is there a clear exit strategy?

Impact: Apart from the criteria above, there are additional commonly applied evaluation criteria such as impact, coverage, connectedness, value-for-money, client satisfaction and protection used in the evaluation, although, not all criteria are applicable to every evaluation. Within the Outcome evaluation there can be additional evaluation questions specified for each the criteria, however all must be agreed with the UNDP in Saudi Arabia. Based on the above analysis, Contractor (herein referred to as evaluation expert) must provide recommendations on how UNDP in Saudi Arabia should adjust its programming, partnership arrangements, resource mobilization strategies, working methods and/or management structures to ensure that the outcome change is achieved by the end of the CPD and beyond.

Methodology: This section suggests an overall approaches and methods for conducting the evaluation, as well as data sources and tools that will likely yield the most reliable and valid answers to the evaluation questions. However, the final decisions about the specific design and methods for the evaluation should emerge from consultations between the evaluation expert and UNDP about what is appropriate and feasible to meet the evaluation purpose, objectives and answers to evaluation questions.

The evaluation expert is encouraged to review the Country Programme Document (CPD) that specifies the outputs, targets and indicators for each component. Based on the objectives and scope mentioned above, the evaluation expert will elaborate a methodology and plan, which will be approved by UNDP and validate information stemmed from contextual sources such as work plans or monitoring reports.

Outcome evaluation will use available data to the greatest extent possible as evidence. This will encompass administrative data as well as various studies and surveys. This approach will help address the possible shortage of data and reveal gaps that should be corrected as the result of the evaluation.

The reliability/availability of disaggregated data should be taken into account as the capacity for data collection at the local level is still developing and it is not possible to conduct comprehensive surveys. In this regard, it is necessary to use objective and subjective data available from the official sources (Central Department of Statistics and Information, data from International Agencies as well as data from various ministries), additionally verified by independent sources such as surveys and studies conducted by local and international research companies, civil society organizations and UN agencies. The relevant sources and access to data will be provided by UNDP and national stakeholders respectively.

The main issues associated with evaluability of some Programme components within Outcome Evaluation might be caused by too general outcome indicators set in the beginning, or their absence. Nonetheless, due to clearly stated overall Country Programme intervention goals and envisaged impact with corresponding indicators there is a certain capacity for data collection, management and analysis in the given Outcome Evaluation. Thus, it is very important to ensure that the Country Programme is evaluable and has an evaluability model that is clearly structured. That, within the model, the goals and objectives are measurable so that the degree to which they have been achieved can be assessed (i.e. answer the question: what data can be collected that will provide clear evidence that the goals and objectives have been met?). In general, the indicators allow the evaluator to ensure that the Country Programme is serving those people it intended to reach, that the relevant data is collected in an organized and consistent fashion.

The Outcome Evaluation will be carried out through a wide participation of all relevant stakeholders including the UNDP and all relevant governmental institutions. Field visits to all project sites; and briefing and debriefing sessions with UNDP, as well as with partners are envisaged. Data collected should be disaggregated (by sex, age and location) whenever possible.

Based on the objectives mentioned above, the evaluation expert will propose a methodology and plan for this assignment, which will be approved by UNDP senior management. An approach relating objectives and/or outcome to indicators, study questions, data required to measure indicators, data sources and collection methods that allow triangulation of data and information often ensure adequate attention is given to all study objectives. However, it’s recommended that the methodology should take into account the following:

The Outcome Evaluation may include, but is not limited to, the following methods of data collection:

Desk review – review and identify relevant sources of information and conceptual frameworks that exist and are available
Examination of contextual information and baselines contained in project documents, National Development Plans (9th and 10th) and Vision 2030, National Transformation Plan 2020, Common Country Assessment, UN Common Country Strategic Framework, Country Programme Document, projects’ documents, progress reports, and other sources. These documents speak to the outcome itself, as opposed to what UNDP is doing about it, and how it was envisaged at certain points in time preceding UNDP’s interventions.
Validation of information about the status of the outcome that is culled from contextual sources such as the CPD, and project progress reports. To do this, consultant may use interviews or questionnaires during the evaluation that seek key respondents’ perceptions on a number of issues, including his/her perception of whether an outcome has changed.
The current status of and degree of change in the outcome shall be assessed against the Country Analysis and the baselines for the outcome and the indicators and benchmarks used in relation to CPD, relevant project documents, progress and monitoring reports of projects/programs, contextual information from partners.
Documents and relevant background material on the development context in Saudi Arabia materials, relevant support documents, evaluations, assessments, and a variety of temporal and focused reports. In particular, the annual reports, respective project documents, project reports, Annual Progress Report (APR) In addition, the evaluation expert could review project budget revisions, progress reports, project files, national strategic and legal documents, and any other materials that the evaluation expert considers useful for this evidence-based assessment.
Undertake a constructive critique of the outcome formulation itself (and the associated indicators). This is integral to the scope of outcome evaluation. The consultants should make recommendations on how the outcome statement can be improved in terms of conceptual clarity, credibility of association with UNDP operations and prospects for gathering of evidence.
Critical analysis of available data (its validity and reliability) with regards to the national guiding documents as well as the intended UNDP inputs to the Government of Saudi Arabia.
Interviews – structured, semi-structured, in-depth, key informant, focus group etc. to capture the perspectives of beneficiaries and non-beneficiaries, participating ministries, departments, relevant personnel from UNDP and local authorities, other relevant stakeholders and others associated with the Country Programme. Interviews with key informants including gathering the information on what the partners have achieved with regard to the outcome and what strategies they have used.
Information systems – analysis of standardized, quantifiable and classifiable regular data linked to a service or process, used for monitoring.
Briefing and debriefing sessions with UNDP and the Government, as well as with partners.

Deliverables of the evaluation:

The evaluation expert will prepare reports which triangulate findings to address the questions of the Outcome evaluation, highlight key significant changes in regard to the key thematic policy documents, draw out lessons learned, present findings and recommendations, reflecting comments and feedback received from selected staff. It is important to receive the report on a timely basis, as reports will be wasted if they arrive too late to inform decisions.

The structure of the reports should be used to guide the reader to the main areas (please, see Annex II for the evaluation report template). It is expected that the reports should include analysis of the outcome pertaining to women and men throughout the report and that gender analysis is not confined to a separate chapter. The reports should be clear, present well-documented and supported findings, and provide concrete and implementable recommendations. UNDP should be able to share it readily with partners and it should generate consensus around the finding and recommendations. The language of the reports should be simple, free from jargon and with specialist terms explained.

Here are the principal evaluation products the evaluation expert is accountable for following activities and deliverables:

Evaluation inception report (submitted with expression of interest and prepared before going into the full-fledged data collection exercise and consist of 5-10 pages excluding annexes) – to clarify the evaluation expert’s understanding of what is being evaluated and why, showing how each evaluation question will be answered by way of: proposed methods, proposed sources of data and data collection procedures (to be presented in an evaluation matrix discussed below). The evaluation inception report should include a proposed schedule of tasks, activities and deliverables. The evaluation inception report provides with an opportunity to verify that all share the same understanding about the evaluation and clarify any misunderstanding at the outset.

Evaluation matrix (suggested as a deliverable to be included in the evaluation inception report) is a tool that evaluation expert creates as map and reference in planning and conducting an evaluation. It also serves as a useful tool for summarizing and visually presenting the evaluation design and methodology for discussions with stakeholders. It details evaluation questions that the evaluation will answer, data sources, data collection, analysis tools or methods appropriate for each data source, and the standard or measure by which each question will be evaluated. (Please, see Table 4 in the attached ToR)

Draft evaluation report (consist of 50-60 pages excluding annexes) – for revision by UNDP Saudi Arabia at the end of data collection. The draft evaluation report should contain all the sections outlined in the Evaluation Report Template (please, see Annex II) and be accompanied by a PowerPoint presentation.

Final evaluation report. The final task of the evaluation expert is to prepare a comprehensive and well-presented copy of the final evaluation report, covering all section of Evaluation Report Template (please, see Annex II) and containing 50-60 pages. Evaluation brief and summary are required. When submitting the final evaluation report, the evaluation expert is required also to provide an "audit tril", detailing how all received comments have (and have not) been addressed in the final evaluation report.

Evaluation plan: The evaluation expert may not begin data collection until the inception report has been reviewed and cleared. The evaluation expert must develop an Evaluation plan and pilot-test the evaluation instruments. The Evaluation plan is a written document that specifies the evaluation design and details its procedures (what needs to be evaluated, with whom, by whom, when, how).

Once approved by UNDP, the Evaluation plan becomes the key management document for the evaluation, guiding delivery in accordance with expectations of UNDP throughout the performance of the contract. The Evaluation plan can have, but is not limited to, the following sections:

Roles and responsibilities of all stakeholders
Evaluation framework
Evaluation calendar
Evaluation criteria
Types of information needed
Sampling and selection of sources of information
Data collection procedures and methods
Methods for analyzing collected information

In preparing Evaluation plan, the evaluation expert is expected to identify what is feasible taking into consideration the time and effort that people involved must contribute.

Supervision and stakeholders’ involvement: In general, the evaluation expert has independence from organizations that have been involved in designing, executing or advising any aspect of the intervention that is the subject of the evaluation. However, UNDP along with Government institutions will have overall responsibility for organizing the Outcome Evaluation and will appoint a focal person/s for coordination in Riyadh. These focal points, with the assistance of UNDP, will backstop and manage the steps involved in planning, implementing and following up the evaluation exercise. On a daily basis, the evaluation expert will work with UNDP and de-brief about the progress of the Outcome Evaluation as needed.

Duty station and logistical modalities: The assignment is home-based with a mission to Saudi Arabia to conduct fieldwork. UNDP will interact with the chosen evaluation expert by communicating through e-mail correspondence while outside of Saudi Arabia, as well as support the evaluation expert in country. There will be an office space, supplies, equipment and materials provided in premises of UNDP.

Evaluation timeframe: The time required will vary depending on the questions the evaluation is attempting to answer, the human and financial resources available, and other external factors. It is important to think through timing issues to ensure that a proposed evaluation is feasible and will provide accurate, reliable, and useful information. It is envisaged that evaluation will take place through April - June 2015 and will involve 35 working days in total (please, see Table 5 in attached ToR).

Competencies

Possess strong analytical skills and the ability to conceptualize, articulate and debate about local governance and human rights issues with a positive and forward-looking attitude;
Understand human rights-based approaches and gender mainstreaming in programming;
Understand results-based management principles, logic modeling/logical framework analysis;
Demonstrate ability to communicate effectively with various partners including government, civil society, private sector, UN Agencies and other development donors;
Excellent organizational and time management skills;
Strong analytical skills and experience in undertaking of similar assignments;
Strong interpersonal skills and ability to work with people from different backgrounds to deliver quality products within a short timeframe;
Excellent report writing skills as well as communication and interviewing skills;
Be flexible and responsive to changes and demands;
Be client oriented and open to feedback.
Required corporate competencies for evaluation expert members:
Sound knowledge of the UNDP programming principles and procedures; the UN system and common country programming processes; the UN evaluation framework, norms and standards; human rights based approach (HRBA);
Demonstrate integrity by modeling the UN’s values and ethical standards;
Promote the vision, mission, and strategic goals of UNDP;
Display cultural, gender, religion, race, nationality and age sensitivity and adaptability;
Fulfill all obligations to gender sensitivity and zero tolerance for sexual harassment.

Required Skills and Experience

Education of evaluation expert members:

MA or PhD in economics, business administration, political science, public policy, development studies, sociology or a related social science.

??Experience of evaluation expert members:

5 or more years of relevant professional experience is required, including previous substantive research experience and involvement in monitoring and evaluation, strategic planning, result-based management.
Experience with quantitative and qualitative data collection and analysis; participatory approaches;
Prior monitoring and evaluation experience in the region is an asset.
Knowledge of the social and political situation and regional development trends in Arab and especially GCC countries is an advantage.

Language Requirements for evaluation expert members:

Proficiency in English language and proven report writing skills, knowledge of Arabic is an asset.