Background
The overall objective of the EU funded “Modernizing Vocational Education and Training (VET) Centres in Azerbaijan” projects is to contribute to the modernization of the vocational education and training system in Azerbaijan, enhancing quality, equality, relevance and access in line with European standards and practices. The projects outputs are to improve quality of education for establishing labor market oriented VET system in Lankaran Economic Region, and to improve quality of education for increasing attractiveness of VET in Ganja city and neighboring regions, as well as increase employability among the youth.
“Support to the establishment of Regional Industrial VET Competence Centre in Ganja”
The specific objective of this action is to improve quality of education to increase attractiveness of VET in Ganja city and neighboring regions, as well as to adjust vocational education to the requirements of the industry. This specific objective will be achieved through strengthening the operational capacities of the Ganja State Vocational Education Centre on Industry and Technology to convert it to a modern Regional VET Competence Centre meeting labor needs of the industry.
“Establishment of Regional VET Centre of Excellence in Lankaran Economic Region”
The specific objective of the action is to improve quality of education through the establishment of labour market oriented VET system in Lankaran Economic Region. The specific objective will be achieved through strengthening the operational capacities of the Jalilabad Vocational Lyceum to convert it to a modern Regional VET Centre of Excellence (CoE) for occupations in agriculture and manufacturing sectors.
It should be noted that both the overall and specific objectives of this action conform to recent policy of the Government of Azerbaijan on diversification of economy through the development of non-oil sector, reduction of disparity between regions and Baku, development of human capital. As seen from the overall and specific objectives, this action will completely conform to the development policy of the Government of Azerbaijan and the projects will cover some of the highlighted challenges, existing problems and shortages of VET system defined in the policy documents.
The evaluator will be requested to evaluate the projects separately, therefore, the evaluation report considering the evaluation criterias and rating table, including the final recommendations will be submitted separately for each project.
Projects has following estimated results under the “Support to the establishment of Regional Industrial VET Competence Centre in Ganja”
ER 1. Regional VET Competence Centre is established
ER 2. Skills and employability of the graduates of VET Competence Centre are enhanced
ER 3. Partnerships with private sector is established
ER 4. Mechanism for recognition/validation of prior learning (competences) is developed and introduced
ER 5. Awareness on EU best practices in VET is increased
Projects has following estimated results under the “Establishment of Regional VET Centre of Excellence in Lankaran Economic Region” projects
ER 1. Regional VET Centre of Excellence is operationalized and modernised
ER 2. Skills and employability of the graduates of VET Competence Centre are enhanced
ER 3. Mechanisms established for partnerships with employers
Duties and Responsibilities
DELIVERABLES
# Deliverable
- Initital desk review and working plan
Description Evaluator clarifies objectives, methods and timing of review
Timing No later than 3 days before the review mission
Responsibilities Evaluator submits to the UNDP CO
- Presentation
Description Initial Findings
Timing End of review mission
Responsibilities Evaluator presents to projects management and UNDP CO
- Draft Final Report
Description Full report (using guidelines on content outlined in Annex B) with annexes
Timing Within 5 days of ending the mission
Responsibilities Sent to the UNDP CO, reviewed by Projects Coordinating Unit, EUD
- Final Report*
Description Revised report with audit trail detailing how all received comments have (and have not) been addressed in the final review report
Timing Within 10 days of receiving UNDP comments on draft
Responsibilities Sent to the UNDP CO
*The final review reports for both projects must be in English. If applicable, the UNDP may choose to arrange for a translation of the report into a language more widely shared by national stakeholders.
IMPLEMENTATION ARRANGEMENTS
The principal responsibility for managing this review resides with the UNDP Country Office (UNDP CO) in Baku, Azerbaijan.
The commissioning unit will contract the consultants and ensure the timely provision of per diems and travel arrangements within the country for the evaluator. The Projects Team will be responsible for liaising with the review evaluator to provide all relevant documents, set up stakeholder interviews, and arrange visits.
OBJECTIVES OF THE FINAL EVALUATION
The final evaluation will assess progress towards the achievement of the projects objectives and outcomes as specified in the Projects Document, and assess the relevance and sustainability of outputs as contributions to mid-term and longer-term outcomes. The evaluation will also review the projects’s strategy, its risks to sustainability.
THE FINAL EVALUATION APPROACH & METHODOLOGY
The evaluation must provide evidence based information that is credible, reliable and useful. The evaluator will review all relevant sources of information including documents prepared during the preparation phase (i.e. UNDP Initiation Plan, the Projects Document, projects reports, projects budget revisions, national strategic and legal documents, and any other materials that the evaluator considers useful for this evidence-based review).
The evaluator is expected to follow a collaborative and participatory approach[1] ensuring close engagement with the UNDP Country Office, EUD, projects team, government counterparts (projects beneficiary) and other key stakeholders. Engagement of stakeholders is vital to a successful evaluation. Stakeholder involvement should include interviews with stakeholders who have projects responsibilities; key experts and consultants in the subject area, Projects team, projects stakeholders, etc.
The final evaluation report should describe the full evaluation approach taken and the rationale for the approach making explicit the underlying assumptions, challenges, strengths and weaknesses about the methods and approach of the review.
DETAILED SCOPE OF THE FINAL EVALUATION
The scope of the final evaluation will cover all activities undertaken in the framework of the projectss. The evaluator will compare planned outputs of the projects to actual outputs and assess the actual results to determine their contribution to the attainment of the projects objective.
The evaluator will assess the following four categories of projects progress.
i. Projects Strategy
Projects design:
. Review the problem addressed by the projectss and the underlying assumptions. Review the effect of any incorrect assumptions or changes to the context to achieving the projectss results as outlined in the Projects Document.
. Review the relevance of the projectss strategy and assess whether it provides the most effective route towards expected/intended results. Were lessons from other relevant projectss properly incorporated into the projects design?
. Review how the projects addresses country priorities. Review country ownership. Was the projects concept in line with the national sector development priorities and plans of the country (or of participating countries in the case of multi-country projectss)?
. Review decision-making processes: were perspectives of those who would be affected by projects decisions, those who could affect the outcomes, and those who could contribute information or other resources to the process, taken into account during projects design processes?
. If there are major areas of concern, recommend areas for improvement.
Results Framework/Logframe:
. Undertake a critical analysis of the projects’s logframe indicators and targets, assess how “SMART” the end-of-projects targets are (Specific, Measurable, Attainable, Relevant, Time-bound).
. Are the projects’s objectives and outcomes or components clear, practical, and feasible within its time frame?
. Examine if progress so far has led to, or could in the future catalyse beneficial development effects (i.e. income generation, gender equality and women’s empowerment, improved governance etc...) that should be included in the projects results framework and monitored on an annual basis.
. Ensure broader development aspects of the projects are being monitored effectively. Develop and recommend SMART ‘development’ indicators.
ii. Progress Towards Results
Progress Towards Outcomes Analysis:
. Review the logframe indicators against progress made towards the end-of-projects targets using the Progress Towards Results Matrix and colour code progress in a “traffic light system” based on the level of progress achieved; assign a rating on progress for each outcome; make recommendations from the areas marked as “Not on target to be achieved” (red).
Table. Progress Towards Results Matrix (Achievement of outcomes against End-of-projects Targets)
Projects Strategy Indicator Baseline Level[2] Level in 1st Narrative session (self- reported) Midterm Target[3]
Objective:
Indicator (if applicable):
ER/Outcome 1: Indicator 1:
Indicator 2:
ER/Outcome 2: Indicator 3:
Indicator 4:
ER/ Outcome 3: Indicator 5:
Indicator 6:
End-of-projects Target Midterm Level & Assessment[4] Achievement Rating[5] Justification for Rating
Objective:
Indicator
(if applicable):
ER/Outcome 1:
Indicator 1:
Indicator 2:
ER/Outcome 2:
Indicator 3:
Indicator 4:
ER/ Outcome 3:
Indicator 5:
Indicator 6:
Indicator Assessment Key
Green= Achieved Yellow= On target to be achieved Red= Not on target to be achieved
In addition to the progress towards outcomes analysis:
. Identify remaining barriers to achieving the projects objective in the remainder of the projects.
. By reviewing the aspects of the projects that have already been successful, identify ways in which the projects can further expand these benefits.
iii. Projects Implementation and Adaptive Management
Management Arrangements:
. Review overall effectiveness of projects management as outlined in the Projects Document. Have changes been made and are they effective? Are responsibilities and reporting lines clear? Is decision-making transparent and undertaken in a timely manner? Recommend areas for improvement.
. Review the quality of execution of the Executing Agency/Implementing Partner(s) (UNDP) and recommend areas for improvement.
. Review the quality of support provided by the Projects Beneficiary (State Customs Committee) and recommend areas for improvement.
Work Planning:
. Review any delays in projects start-up and implementation, identify the causes and examine if they have been resolved.
. Are work-planning processes results-based? If not, suggest ways to re-orientate work planning to focus on results?
. Examine the use of the projects’s results framework/ logframe as a management tool and review any changes made to it since projects start.
Finance and co-finance:
. Consider the financial management of the projects, with specific reference to the cost-effectiveness of interventions.
. Review the changes to fund allocations as a result of budget revisions and assess the appropriateness and relevance of such revisions.
. Does the projects have the appropriate financial controls, including reporting and planning, that allow management to make informed decisions regarding the budget and allow for timely flow of funds?
. Informed by the co-financing monitoring table to be filled out, provide commentary on co-financing: is co-financing being used strategically to help the objectives of the projects? Is the Projects Team meeting with all co-financing partners regularly in order to align financing priorities and annual work plans?
Projects-level Monitoring and Evaluation Systems:
. Review the monitoring tools currently being used: Do they provide the necessary information? Do they involve key partners? Are they aligned or mainstreamed with national systems? Do they use existing information? Are they efficient? Are they cost-effective? Are additional tools required? How could they be made more participatory and inclusive?
. Examine the financial management of the projects monitoring and evaluation budget. Are sufficient resources being allocated to monitoring and evaluation? Are these resources being allocated effectively?
Stakeholder Engagement:
. Projects management: Has the projects developed and leveraged the necessary and appropriate partnerships with direct and tangential stakeholders?
. Participation and country-driven processes: Do local and national government stakeholders support the objectives of the projects? Do they continue to have an active role in projects decision-making that supports efficient and effective projects implementation?
. Participation and public awareness: To what extent has stakeholder involvement and public awareness contributed to the progress towards achievement of projects objectives?
Reporting:
. Assess how adaptive management changes have been reported by the projects management and shared with the Projects Board.
. Assess how well the Projects Team and partners undertake and fulfil reporting requirements.
. Assess how lessons derived from the adaptive management process have been documented, shared with key partners and internalized by partners.
Communications:
. Review internal projects communication with stakeholders: Is communication regular and effective? Are there key stakeholders left out of communication? Are there feedback mechanisms when communication is received? Does this communication with stakeholders contribute to their awareness of projects outcomes and activities and investment in the sustainability of projects results?
. Review external projects communication: Are proper means of communication established or being established to express the projects progress and intended impact to the public (is there a web presence, for example? Or did the projects implement appropriate outreach and public awareness campaigns?)
. For reporting purposes, write one half-page paragraph that summarizes the projects’s progress towards results in terms of contribution to sustainable development benefits, as well as to global benefits.
iv. Sustainability
. Validate whether the risks identified in the Projects Document, Narrative reports are the most important and whether the risk ratings applied are appropriate and up to date. If not, explain why.
. In addition, assess the following risks to sustainability:
Financial risks to sustainability:
. What is the likelihood of financial and economic resources not being available once the EU grant assistance ends (consider potential resources can be from multiple sources, such as the public and private sectors, income generating activities, and other funding that will be adequate financial resources for sustaining projects’s outcomes)?
Socio-economic risks to sustainability:
. Are there any social or political risks that may jeopardize sustainability of projects outcomes? What is the risk that the level of stakeholder ownership (including ownership by governments and other key stakeholders) will be insufficient to allow for the projects outcomes/benefits to be sustained? Do the various key stakeholders see that it is in their interest that the projects benefits continue to flow? Is there sufficient public / stakeholder awareness in support of the long term objectives of the projects? Are lessons learned being documented by the Projects Team on a continual basis and shared/ transferred to appropriate parties who could learn from the projects and potentially replicate and/or scale it in the future?
Institutional Framework and Governance risks to sustainability:
. Do the legal frameworks, policies, governance structures and processes pose risks that may jeopardize sustenance of projects benefits? While assessing this parameter, also consider if the required systems/ mechanisms for accountability, transparency, and technical knowledge transfer are in place.
Conclusions & Recommendations
The evaluator will include a section of the report setting out the evaluation’s evidence-based conclusions, in light of the findings.
Recommendations should be succinct suggestions for critical intervention that are specific, measurable, achievable, and relevant.
The evaluator should make no more than 10 recommendations total.
Ratings
The evaluator will include the ratings of the projects’s results and brief descriptions of the associated achievements in Evaluation Ratings & Achievement Summary Table in the evaluation report. See Annex E for ratings scales. No rating on Projects Strategy and no overall projects rating is required.
Table. Evaluation Ratings & Achievement Summary Table for Projects
Measure Rating Achievement Description Projects Strategy N/A Progress Towards ResultsObjective Achievement Rating: (rate 6 pt. scale) Outcome 1 Achievement Rating: (rate 6 pt. scale) Outcome 2 Achievement Rating: (rate 6 pt. scale) Outcome 3 Achievement Rating: (rate 6 pt. scale) Etc. Projects Implementation & Adaptive Management (rate 6 pt. scale) |
Sustainability (rate 4 pt. scale)
TIMEFRAME
The total duration of the final evaluation will be approximately (1 month) starting from February 2020 to March 2020. Total number of working days: 30. The tentative evaluation timeframe is as follows:
TIMEFRAME ACTIVITY
February 4 – February 10, 2019 Project documents review and working plan accordingly
February 12-22, 2020 Evaluation mission in Baku and in the regions: 10 days for both
projects (5 days in each)
Stakeholder/partner meetings, interviews
Debrief and briefing with the UNDP relevant Staff
February 24, 2020 – March 5, 2020 Preparing and submitting of draft evaluation report
March 12 - 20, 2020 Finalization of Evaluation report and Submission of Final Report to UNDP CO[6]
For ideas on innovative and participatory Monitoring and Evaluation strategies and techniques, see UNDP Discussion Paper: Innovations in Monitoring & Evaluating Results, 05 Nov 2013.
Populate with data from the Projects Document
If available
Colour code this column only, if available
Use the 6 point Progress Towards Results Rating Scale: HS, S, MS, MU, U, HU
Please note that the evaluation report considering the evaluation criterias and rating table, including the final recommendations will be submitted separately for each project.
Competencies
Qualifications and Competencies:One independent evaluator will perform the evaluation. The evaluator should have prior experience in reviewing or evaluating similar projectss. Experience with UNDP/EU financed projectss is an advantage. The selection of consultants will be aimed at maximizing the overall qualities in the following areas:
- At least university degree in a relevant discipline (e,g. social sciences, education, economics, law and other field relevant to ToR ;
- -Professional experience (minimum 7 years) with result-based management evaluation methodologies applying SMART indicators and evaluation/review experiences within UN/EU system;
- - Appproach to work indicating how this evaluaton will be conducted
- - Skype interview
- Excellent communication skills (written and spoken English; Russian will be an asset );
- Demonstrable analytical skills.
Required Skills and Experience
Education:
At least university degree in a relevant discipline (e,g. social sciences, education, economics, law and other field relevant to ToR ; (15 points)
Experience:
Professional experience (minimum 7 years) with result-based management evaluation methodologies applying SMART indicators and evaluation/review experiences within UN/EU system; (20 points, max points will be given to the candidate with relevant experiences covering all aspects indicated under the particular evaluation criteria)
Languages:
Excellent communication skills (written and spoken English; Russian will be an asset )
Other requirements:
Appproach to work indicating how this evaluaton will be conducted (15 points)
Skype interview- (20 points)
Demonstrable analytical skills
Criteria for Evaluation of Proposal:
Individual consultants will be evaluated based on the following methodologies:
1. Cumulative evaluation that takes into account both financial offer and the technical expertise of the potential candidates
A two-stage procedure will be utilized in evaluating the proposals, with evaluation of the technical component being completed prior to any price component being reviewed and compared. The Price Component will be reviewed only for those individuals whose Technical Component meets the requirements for the assignment. The total number of points which individual may obtain for both components is 100.
The technical component, which has a total possible value of 70 points, will be evaluated using the following criteria:
- At least university degree in a relevant discipline (e,g. social sciences, education, economics, law and other field relevant to ToR ; (15 points)
-Professional experience (minimum 7 years) with result-based management evaluation methodologies applying SMART indicators and evaluation/review experiences within UN/EU system; (20 points, max points will be given to the candidate with relevant experiences covering all aspects indicated under the particular evaluation criteria)
- Appproach to work indicating how this evaluaton will be conducted (15 points)
- Skype interview- (20 points)
- Excellent communication skills (written and spoken English; Russian will be an asset)
- Demonstrable analytical skills.
If the substantive presentation of a technical proposal achieves the minimum of 49 points, the competitiveness of the offered consultancy expenses will be taken into account in the following manner:
The total amount of points for the fees component is 30. The maximum number of points shall be allotted to the lowest fees proposed that is compared among those invited individuals which obtain the threshold points in the evaluation of the technical proposal. All other proposals shall receive points in inverse proportion to the lowest fees; e.g.;
[ 30 Points] x [US$ lowest]/ [US$ other] = points for other proposer’s fees
The work and performance of International Consultant/Expert on Projects Final Evaluation will be monitored by the UNDP Programme Advisor and Projects Officer to ensure that all required activities are on-course and on-schedule. The key factor of the monitoring will be the quality and eligibility of the deliverables. During the contract period the working days: 30 consultancy days.
ToR ANNEX A: List of Documents to be reviewed by the evaluator
- UNDP Projects Document
- All Projects Implementation Reports (annual reporting to EU, Narrative sessions)
- Some consultants reports, if requested
- Oversight mission reports
- Financial and Administration guidelines used by Projects Team
The following documents will also be available:
- Programme and Operations Policies and Procedures
- UNDP country/countries programme document(s) and others
ToR ANNEX B: Guidelines on Contents for the Final Evaluation Report[1]
i. Basic Report Information (for opening page or title page)
. Title of UNDP supported EU financed projects
. UNDP projects ID#
. Evaluation time frame and date of evaluation report
. Region and countries included in the projects
. Executing Agency/Implementing Partner and other projects partners
. Acknowledgements
ii. Table of Contents
iii. Acronyms and Abbreviations
1. Executive Summary (3-5 pages)
. Projects Information Table
. Projects Description (brief)
. Projects Progress Summary (between 200-500 words)
. Evaluation Ratings & Achievement Summary Table
. Concise summary of conclusions
. Recommendation Summary Table
2. Introduction (2-3 pages)
. Purpose of the Final Evaluation and objectives
. Scope & Methodology: principles of design and execution of the evaluation, evaluation approach and data collection methods, limitations to the final evaluation
. Structure of the Final Evaluation Report
3. Projects Description and Background Context (3-5 pages)
. Development context: socio-economic, institutional, and policy factors relevant to the projects objective and scope
. Problems that the projects sought to address: threats and barriers targeted
. Projects Description and Strategy: objective, outcomes and expected results, description of field sites (if any)
. Projects Implementation Arrangements: short description of the Projects Board, key implementing partner arrangements, etc.
. Projects timing and milestones
. Main stakeholders: summary list
4. Findings (12-14 pages)
4.1
Projects Strategy
. Projects Design
. Results Framework/Log frame
4.2 Progress Towards Results
. Progress towards outcomes analysis
. Remaining barriers to achieving the projects objective
4.3 Projects Implementation and Adaptive Management
. Management Arrangements
. Work planning
. Finance and co-finance
. Projects-level monitoring and evaluation systems
. Stakeholder engagement
. Reporting
. Communications
4.4 Sustainability
. Financial risks to sustainability
. Socio-economic to sustainability
. Institutional framework and governance risks to sustainability
. Risks to sustainability
5. Conclusions and Recommendations (4-6 pages)
5.1
Conclusions
. Comprehensive and balanced statements (that are evidence-based and connected to the evaluation findings) which highlight the strengths, weaknesses and results of the projects
5.2 Recommendations
. Corrective actions for the design, implementation, monitoring and evaluation of the projects
. Actions to follow up or reinforce initial benefits from the projects
. Proposals for future directions underlining main objectives
6. Annexes
. Final Evaluation ToR (excluding ToR annexes)
. Final evaluative matrix (evaluation criteria with key questions, indicators, sources of data, and methodology)
. Example Questionnaire or Interview Guide used for data collection
. Ratings Scales
. Evaluation mission itinerary
. List of persons interviewed
. List of documents reviewed
. Co-financing table (if not previously included in the body of the report)
. Signed UNEG Code of Conduct form
. Signed evaluation final report clearance form
ToR ANNEX C: Evaluative Matrix Template
Projects Strategy: To what extent is the projects strategy relevant to country priorities, country ownership, and the best route towards expected results?
Evaluative Questions (include evaluative question(s))
Indicators (i.e. relationships established, level of coherence between projects design and implementation approach, specific activities conducted, quality of risk mitigation strategies, etc.) Sources (i.e. projects documents, national policies or strategies, websites, projects staff, projects partners, data collected throughout the evaluation mission, etc.)
Methodology (i.e. document analysis, data analysis, interviews with projects staff, interviews with stakeholders, etc.)
Progress Towards Results: To what extent have the expected outcomes and objectives of the projects been achieved thus far?
Evaluative Questions
Indicators
Sources
Methodology
Projects Implementation and Adaptive Management: Has the projects been implemented efficiently, cost-effectively, and been able to adapt to any changing conditions thus far? To what extent are projects-level monitoring and evaluation systems, reporting, and projects communications supporting the projects’s implementation?
Evaluative Questions
Indicators
Sources
Methodology
Sustainability: To what extent are there financial, institutional, socio-economic, and/or other risks to sustaining long-term projects results?
Evaluative Questions
Indicators
Sources
Methodology
The Report length should not exceed 40 pages in total (not including annexes).
The total duration of the final evaluation will be approximately (1 month) starting from February 2020 to March 2020. Total number of working days: 30. Evaluation mission in Baku and in the regions: 10 days for both projects (5 days in each).
The International Consultant has to provide:
- Offeror’s letter to UNDP conforming interest and availability for the Individual Contractor (IC)
Assignment and Breakdown of Costs (in USD) Supporting the Final All-Inclusive Price;
- Filled out and signed P11 Form or CV;
- 2 reference letters.
The system accepts only one attachment, so please send both technical and financial proposals as one attachment.