Background
Sustainable Road and Transport Management
Terms of Reference
Final Evaluation
June 2021
- Background and context
The National Transport Strategy (NTS) for 2030 was developed to help fulfil Kingdom Vision 2030 by identifying the challenges faced by government entities and establishing targets and initiatives to overcome those challenges and the Project is participating in achieving the NTS, set targets and Kingdom Vision 2030 strategic objectives which will lead to achieve the Sustainable Development Goals especially Goal (9) ‘‘Industry, Innovation and Infrastructure’’ and Goal (13) ‘‘Climate Action’’ where these two goals are critical drivers of economic growth and development of KSA road network.
Therefore, the tasks related to both the Vision 2030 and the National Transportation Strategy (NTS2030) initiatives require a high level of coordination among Ministry of Transport (MOT) departments and with other government agencies and demand additional specialized knowledge to support the Strategic Planning Department and other concerned MOT entities. Due to the new context of the Vision 2030 and the current need for MOT to focus on the implementation of the new initiatives such as coordination with Ministry of Municipal Rural Affairs MOMRA, therefore, the duration of the Project needs to be extended beyond the original to 2022.
Basic Project information can also be included in table format as follows:
PROJECT/OUTCOME INFORMATION | ||
Project/outcome title | Sustainable Road and Transport Management | |
Atlas ID | SAU10/79238 | |
Corporate outcome and output | Public sector strengthened through improved efficiency, effectiveness, equity and accountability | |
Country | Saudi Arabia | |
Region | RBAS | |
Date project document signed |
| |
Project dates | Start | Planned end |
01/01/2012 | 31/12/2022 | |
Project budget | 16.2M | |
Estimated Project expenditure at the time of evaluation | 13M | |
Funding source | Government | |
Implementing party | Ministry of Transport |
Duties and Responsibilities
2. Evaluation purpose, scope and objectives
The project has been ongoing since 2011 and has, thus far, never been evaluated. Drastic changes have been taking place in the country and the project has had to adapt to the changes over recent years, this included changes in Ministers, Deputy Ministers and Project staff, resulting in changing project directions. In order to ensure the project has delivered its intended objectives and to provide recommendations for the way forward, it was imperative to conduct a final evaluation. This evaluation will benefit the Ministry of Transport in their planning for future years to meet Saudi Vision 2030 and highlight the impacts this project has had on the transport sector over the past few years.
This evaluation should cover all components of the project (such as road safety initiatives, trade facilitation and etc), those relevant before Saudi Vision 2030 and those after and its impact on the sector in the whole country. However, the evaluator shall consider the Saudi Vision 2030 objective’s and other MOT Initiatives in formulating the new Project Document which can both improve the sustainability of benefits from this project, and aid in the overall enhancement of UNDP Goals as mentioned above.
Evaluation criteria and key guiding questions
Evaluation questions define the information that this evaluation will generate. Questions should be grouped according to the four OECD-DAC evaluation criteria: (a) relevance; (b) effectiveness; (c) efficiency (d) impact; and (e) sustainability (and/or other criteria used).
The mainstream definitions of the OECD-DAC criteria are neutral in terms of human rights and gender dimensions and these dimensions need to be added into the evaluation criteria chosen (see page 77, table 10 of Integrating Human Rights and Gender Equality in Evaluations).
Ratings must be provided on the following performance criteria. The completed table must be included in the evaluation executive summary.
Evaluation Ratings: | |||
1. Monitoring and Evaluation | rating | 2. IA& EA Execution | rating |
M&E design at entry |
| Quality of UNDP Implementation |
|
M&E Plan Implementation |
| Quality of Execution - Implementing Partner |
|
Overall quality of M&E |
| Overall quality of Implementation |
|
3. Assessment of Outcomes | rating | 4. Sustainability | rating |
Relevance |
| Financial resources: |
|
Effectiveness |
| Socio-political: |
|
Efficiency |
| Institutional framework and governance: |
|
Impact |
| Sustainability |
|
Overall Project Outcome Rating |
| Environmental : |
|
|
| Overall likelihood of sustainability: |
|
Relevance
Effectiveness
Efficiency
Impact:
Sustainability
|
| |
|
|
Evaluation cross-cutting issues sample questions
Human rights
Gender equality
Guiding evaluation questions can be further refined by the evaluator and agreed with UNDP/Project management and the- evaluation stakeholders in the inception report.
Evaluation Approach and Methodology
An overall approach and method[1] for conducting project terminal evaluations of UNDP. The methodology of work will consist of desk review of relevant project documentation and direct consultations with the project management, staff and other key local stakeholders during two weeks site visit to Riyadh in September2021. The overall duration of the assignment is expected to consist of a site visit of 15 days includes a corresponding amount of desk work to pre-review the required project documentation and to finalize the reporting. This makes the total working days to be 21working days, the timeframe detailed shall be as stipulated below. In carrying out the evaluation task, the consultant will pay particular attention to the following:
The evaluator must use gender-based methodology and tools and ensure that gender equality and women's empowerment, inclusion of vulnerable groups as well as other cross-cutting issues and the SDGs, are included in the final evaluation report and the new project document.
The evaluation must provide factual information that is credible, reliable and useful. The final methodological approach including interview schedule, field visits and data to be used in the evaluation should be clearly outlined in the inception report and be fully discussed and agreed between UNDP, stakeholders and the evaluators.
Evaluation products (deliverables)
An evaluation report and an associated power point presentation summarizing the findings of the evaluation and the proposed follow-up actions in a new UNDP Project Document format.
The consultant will be expected to deliver the following:
The consultant should present three hard copies of the report as well as an electronic copy. The draft final report should be submitted not later than three weeks after the end of the on-site mission and the final report and the Project document within two weeks from receiving the comments of the project management and UNDP on the draft reports. The Project document will be formulated with technical assistance provided by the Ministry and the Project Manager.
Standard templates that need to be followed are provided in the Annexes section. It is expected that the evaluator will follow the UNDP evaluation guidelines and UNEG quality check list and ensure all the quality criteria are met in the evaluation report.
In line with UNDP’s financial regulations, when determined by the Country Office and/or the consultant that a deliverable or service cannot be satisfactory completed due to impact of COVID-19 and limitations to the evaluation, that deliverable or service will not be paid. Due to the current COVID-19 situation and its implications, a partial payment may be considered if the consultant invested time towards the deliverable but was unable to complete to circumstances beyond his/her/their control.
[1] For additional information on methods, see the Handbook on Planning, Monitoring and Evaluating for Development Results, Chapter 7, pg. 163 |
Competencies
Evaluation consultant required competencies
The consultancy will be conducted by one individual evaluator (national or international).
The evaluator shall have prior experience in evaluating similar projects and have sufficient experience in transport sector planning related to initiatives and main national and international development operations related to road sector specifically.
The evaluator selected should have not participate in the project implementation in order to avoid any conflict of interest with the project related activities.
The evaluator must present the following qualifications:
- Advanced university degree in transport planning and engineering and obtain Professional Transport Planner (PTP) certificate is preferable;
- Minimum 15 years of relevant professional experience in the area of transport planning and initiatives;
- Familiar with UNDP approaches to development assistance;
- Demonstrated understanding of issues related to human rights and gender; experience in gender sensitive evaluation and analysis in a development project; Previous experience with results-based formulating, monitoring and evaluation methodologies;
- Strong communication and analytical skills;
- Strong command of English language, both written and spoken;
- Previous work experience in the region is an asset.
Evaluation ethics
This evaluation will be conducted in accordance with the principles outlined in the UNEG ‘Ethical Guidelines for Evaluation’. The consultant must safeguard the rights and confidentiality of information providers, interviewees and stakeholders through measures to ensure compliance with legal and other relevant codes governing collection of data and reporting on data. The consultant must also ensure security of collected information before and after the evaluation and protocols to ensure anonymity and confidentiality of sources of information where that is expected. The information knowledge and data gathered in the evaluation process must also be solely used for the evaluation and not for other uses with the express authorization of UNDP and partners.”
Implementation arrangements
UNDP Saudi Arabia Country Office will select the consultant through a transparent process in consultation with MOT. UNDP will be responsible for the management of the consultant and will in this regard designate an evaluation manager. The project management unit will assist in facilitating the process (e.g., providing relevant documentation, arranging visits/interviews with key informants, participate in reviewing the evaluation deliverables and arrange for the consultant all necessary site visits and meetings in Saudi Arabia according to the ToR). UNDP country office in coordination with the project management unit shall arrange logistics for the mission including hotel reservation and transportation during the mission.
The evaluation manager will convene an evaluation reference group comprising of technical experts from UNDP, donors and implementing partners. This reference group will review the inception report and the draft review report to provide detailed comments related to the quality of methodology, evidence collected, analysis and reporting. The reference group will also advise on the conformity of processes to the UNDP evaluation guidelines. The evaluator needs to show how he/she addressed the comments.
The consultant will take responsibility, with assistance from the project manager, for conducting the meetings and the review, subject to advanced approval of the methodology submitted in the inception report. Project staff will not participate in the meetings between the consultant and the evaluation participants.
The consultant will report directly to the designated evaluation manager and work closely with the project manager.
The final report and project document will be approved by the evaluation commissioner.
UNDP with support of relevant stakeholders will develop a management response to the evaluation within 2 weeks of report finalization.
Required Skills and Experience
Time frame for the evaluation process
Activity | Timing/working days | Estimated Completion Date | Note |
Desk Review | 3 | 15 September 2021 | In home country |
Evaluation Mission | 5 | 20 September 2021 | In KSA |
Draft Evaluation Report | 5 | 30 September 2021 |
In home country |
Final Report | 2 | 10 October 2021 | |
Draft Project Document | 4 | 17 October 2021 | In home country |
Final Project Document | 2 | 24 October 2021 | |
Total | 21 | September – October 2021 |
EVALUATION DELIVERABLES
The Evaluator is expected to deliver the following:
Deliverable | Content | Timing | Responsibilities |
Inception Report | Evaluator provides clarifications on timing and method | On arrival Day (15 September 2021) | Evaluator submits to Project Management and UNDP CO. |
Debriefing presentations | Initial Findings | End of evaluation mission (19 September 2021) | To Project Management and UNDP CO. |
Draft Final Report | Full report, (per annexed template) with annexes | Within 2 weeks of the evaluation mission (30 September 2021) | Sent to Project Management and UNDP CO, reviewed by MOT. |
Final Report* | Revised report | Within 1 week of receiving UNDP comments on draft (10 October 2021) | To Project Management and UNDP CO. |
Draft Project Document | Full Project Document | Within 4 weeks of the evaluation mission (17 October 2021) | Sent to Project Management and UNDP CO, reviewed by MOT. |
Final Project* Document | Revised Project Document | Within 1 week of receiving UNDP comments on draft (24 October 2021) | To Project Management and UNDP CO. |
*When submitting the final evaluation report and final Project Document, the evaluator is required also to provide an 'audit trail', detailing how all received comments have (and have not) been addressed in the final evaluation report and the Project Document.
Payment modalities and specifications
% | Milestone |
20% | Acceptance of Inception Report prior to the field visit. |
30% | Following approval of the final Evaluation Report. |
50% | Following approval of the final Project Document. |
In accordance with UNDP's financial regulations, where the country office and/or the consultant determine that a deliverable or service cannot be provided satisfactorily due to the impact of COVID-19 and limitations on evaluation, that deliverable or service will not be paid.
Due to the current situation and the implications of COVID-19, a partial payment may be considered if the consultant has invested time in the production of the deliverable but has not been able to ensure its full supply due to circumstances beyond his control.
Estimated working day allocation and schedule for this evaluation
ACTIVITY | ESTIMATED # OF WORKING DAYS | DATE OF COMPLETION | PLACE | RESPONSIBLE PARTY |
Phase One: Desk review and inception report | ||||
Meeting briefing with UNDP (programme managers and project staff as needed) | - | At the time of the Mission Started 15 September 2021 | UNDP or remote | Evaluation manager and commissioner |
Sharing of the relevant documentation with the evaluation team | - | At the time of contract signing 8 September 2021 | Via email | Evaluation manager and commissioner |
Desk review, Evaluation design, methodology and updated workplan including the list of stakeholders to be interviewed | 3 days | Within two weeks of contract signing 15 September 2021 | Home- based | Evaluation Consultant |
Submission of the inception report (15 pages maximum) | - | By 15 September 2021 |
| Evaluation Consultant |
Comments and approval of inception report | - | By 17 September 2021 | UNDP | Evaluation manager |
Phase Two: Data-collection mission | ||||
Consultations and field visits, in-depth interviews and focus groups | 5 | 19 September 2021 | In country
With field visits | UNDP to organize with local project partners, project staff, local authorities, etc. |
Debriefing to UNDP and key stakeholders |
| 19 September 2021 | In country | Evaluation Consultant |
Phase Three: Evaluation report writing | ||||
Preparation of draft evaluation report (50 pages maximum excluding annexes), executive summary (5 pages) and Draft Report submission | 5 | Within Two weeks of the completion of the field mission End of September 2021 | Home- based | Evaluation Consultant |
Consolidated UNDP and stakeholder comments to the draft report | - | By 6 October 2021 | UNDP | Evaluation manager and evaluation reference group |
Finalization and submission of the evaluation report incorporating additions and comments provided by project staff and UNDP country office | 2 | by 10 October 2021 | Home- based | Evaluation Consultant |
Preparation of the draft Project Document | 4 | Within one week of final evaluation report acceptance 17 October 2021 | Home- based | Evaluation Consultant |
Consolidated UNDP and stakeholder Comments on Project Document | - | Within one week from draft PD submission 23 October 2021 |
| UNDP and National Partner |
Finalization of PD and submission | 2 | Within 5 days from receiving comments 30 October 2021 |
| Evaluation Consultant |
Estimated total days for the evaluation | 21 |
|
|
|
- Application submission process and criteria for selection
As required by the programme unit.
- TOR annexes
Annexes can be used to provide additional detail about evaluation background and requirements to facilitate the work of evaluators. Some examples include:
- Intervention results framework and theory of change. Provides more detailed information on the intervention being evaluated.
- Key stakeholders and partners. A list of key stakeholders and other individuals who should be consulted, together with an indication of their affiliation and relevance for the evaluation and their contact information. This annex can also suggest sites to be visited.
- Documents to be consulted. A list of important documents and web pages that the evaluators should read at the outset of the evaluation and before finalizing the evaluation design and the inception report. This should be limited to the critical information that the evaluation team needs. Data sources and documents may include:
- Relevant national strategy documents,
- Strategic and other planning documents (e.g., programme and project documents).
- Monitoring plans and indicators.
- Partnership arrangements (e.g., agreements of cooperation with Governments or partners).
- Previous evaluations and assessments.
- UNDP evaluation policy, UNEG norms and standards and other policy documents.
- Evaluation matrix (suggested as a deliverable to be included in the inception report). The evaluation matrix is a tool that evaluators create as map and reference in planning and conducting an evaluation. It also serves as a useful tool for summarizing and visually presenting the evaluation design and methodology for discussions with stakeholders. It details evaluation questions that the evaluation will answer, data sources, data collection, analysis tools or methods appropriate for each data source, and the standard or measure by which each question will be evaluated.
Table 1. Sample evaluation matrix
Relevant evaluation criteria | Key questions | Specific sub questions | Data sources | Data-collection methods/tools | Indicators/ success standard | Methods for data analysis |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- Schedule of tasks, milestones and deliverables. Based on the time frame specified in the TOR, the evaluators present the detailed schedule.
- Required format for the evaluation report. The final report must include, but not necessarily be limited to, the elements outlined in the quality criteria for evaluation reports (). The standard template can be found in the following link (Evaluation report).
- Code of conduct. UNDP programme units require each member of the evaluation team to read carefully, understand and sign the ‘Code of Conduct for Evaluators in the United Nations system’, it should be made available as an attachment to the evaluation report.
- Inception report
- Audit trail
- UNDP Evaluation Guidelines
- Evaluation Quality Assessment (the final report should address all the questions in the checklist pages 8- 11)
Interested individual consultants must submit the following documents/information to demonstrate their qualifications in one single PDF document:
- Technical Methodology/Proposal
- Duly accomplished Confirmation of Interest and Submission of Financial Proposal Template using the template provided by UNDP (Annex II);
- Personal CV or P11, indicating all past experience from similar projects, as well as the contact details (email and telephone number) of the Candidate and at least three (3) professional references.
The interested offeror must read the Individual Consultant (IC) Procurement Notice, which can be viewed at http://procurement-notices.undp.org/view_notice.cfm?notice_id=62795 for more detail information about term of references, instructions to the offeror, and to download the documents to be submitted in the offer through online.
UNDP reserves rights to reject any applications that are incomplete.
Please be informed that we don’t accept application submitted via email.
Interested Offerors are required to submit an application via UNDP Jobsite system as the application screening and evaluation will be done through UNDP Jobsite system. Please note that UNDP Jobsite system allows only one uploading of application document, so please make sure that you merge all your documents into a single file. Your on¬line application submission will be acknowledged where an email address has been provided. If you do not receive an e¬mail acknowledgement within 24 hours of submission, your application may not have been received. In such cases, please resubmit the application, if necessary. Please combine all your documents into one (1) single PDF document as the system only allows to upload maximum one document.
Any request for clarification/additional information on this procurement notice shall be communicated in writing to UNDP office or send to email mohammed.abbas@undp.org. While the Procurement Unit would endeavour to provide information expeditiously, only requests receiving at least 3 working days prior to the submission deadline will be entertained. Any delay in providing such information will not be considered as a reason for extending the submission deadline. The UNDP's response (including an explanation of the query but without identifying the source of inquiry) will be posted in the Individual Consultant (IC) Procurement Notice page as provided above. Therefore, all prospective Offerors are advised to visit the page regularly to make obtain update related to this Individual Consultant (IC) Procurement Notice
UNDP is committed to achieving workforce diversity in terms of gender, nationality and culture. Individuals from minority groups, indigenous groups and persons with disabilities are equally encouraged to apply. All applications will be treated with the strictest confidence.
UNDP does not tolerate sexual exploitation and abuse, any kind of harassment, including sexual harassment, and discrimination. All selected candidates will, therefore, undergo rigorous reference and background checks.
Annexes:
Annex-1 – Template Confirmation of Interest and Submission of Financial Proposal.
Annex-2 – Individual Consultant General Terms and Conditions.
Annex 3 : TOR
Forms are available on the following link
https://www.dropbox.com/sh/zwnm3ns28iavyeh/AAA7bhxAjH5lxAWmiNnJBjksa?dl=0