Background

In line with the five priorities of the UNDAF 2007-2011, developed after an extensive process of consultations with the Government and other stakeholders in the country, UNDP identified three interrelated key areas to be addressed during the Country Programme Document (CPD) period (2007-2011). They come under the umbrella of support for creating an enabling environment for achieving the MDGs as the Country Office (CO) continues to work in assisting the Government and people of Ethiopia to make progress in achieving their goals. The three areas are: reducing poverty and working towards achievement of the MDGs; deepening democratic governance; and addressing the HIV/AIDS epidemic. Working with the Government as well as other United Nations bodies, UNDP would support the development of at least one economic growth corridor with potential interventions related to diversification in agriculture, industry and services to increase the income of the poor. Support will be provided in the form of knowledge-building, increased labour productivity, and encouraging greater use of technology.

Following to the Country Programme Mid Term Evaluation (2009), UNDP Ethiopia programme went through a country programme re-profiling. The programme is currently framed around three strategic priorities: enhanced economic growth and poverty reduction; democratic governance and capacity development; and development of a low-carbon and climate-resilient economy (LCCR). Under programme component I: Enhanced Economic Growth and Poverty Reduction UNDP will support the Government’s efforts to accelerate pro-poor economic growth, with agricultural and small- and medium-scale enterprises as the key drivers, anchored in a value-chain approach. The focus will be on supporting policy and institutional capacity development at both the national and sub-national levels to build the knowledge, skills and systems that can enhance access to critical productive services, support efficiency of marketing systems, and leverage appropriate technology and practices to demonstrate sustainable options for boosting productivity and income in rural areas. UNDP will support policy and diagnostic work, knowledge networking, codification of knowledge, and testing of innovative practices that can facilitate solutions to practical bottlenecks in building effective value chains; and strengthen capacities of producer and private sector institutions to enhance access of the poor, especially women and youth, to better technologies, inputs, finance and markets. 

Before the re-profiling of the UNDP programme, the outcome identified under the UNDP Country Programme Action Plan component relative to Enhanced Economic Growth and Poverty Reduction was the following: Knowledge about Economic Growth Corridors (EGCs) enhanced and private sector development promoted as an engine of Economic Growth. After the re-profiling the outcome statement became:  By end 2011, at least one EGC is identified based on the national strategic framework and ready to be operational. Moreover, after the programme re-profiling a new outcome addressing private sector development was formulated:Private Sector Development promoted as a motor of Economic Growth.

Outcome indicators covering the cycle 2007-2011:

  • Number of EGS defined with sound investment plan based on national framework and gender responsive in-depth preparatory studies;
  • Extent of documented knowledge on economic growth corridors in Ethiopia;
  • Investment plan available and the number of new productive initiatives that promote pro-poor growth implemented within the EGC;
  • Number of new private domestic and FD investments as a result of improved investment climate (from 2010);
  • Level of  competitiveness of private enterprise (2010).
UNDAF Outcome

By 2011, at national, regional, organizational and business levels, capacity strengthened and knowledge developed for increased incomes of the poor, through enhanced labor factor productivity and more intensive and widespread use of technology in at least one economic growth corridor, with potential interventions related to expansion and diversification in agriculture, industry and services.

National Goal
Accelerating Commercialization of agriculture and private sector development to have an accelerated and sustained economic growth for eradication of poverty and improve people's livelihood.

Purpose and Objective

The Purpose of the outcome evaluation is to enhance development effectiveness, to assist decision making, to assist policy making, to re-direct future UNDP assistance.

The overall objective of the outcome-level evaluation is to find out how UNDP has gone about supporting processes and building capacities that have, indeed, helped make a difference. In doing so, evaluation aims to identify which approaches have worked well and which have faced challenges, and to use lessons learned to improve future initiatives. This evaluation also serves the purpose of holding UNDP accountable for the resources invested in its work.
 
The outcome evaluation will be undertaken to learn lessons and good practices so that this knowledge can contribute to the next Programme Cycle 2012-2015. To learn lessons for next CP formulation by identifying the status of outcome and factors affecting it, relevance of outcome/outputs, strategic positioning of UNDP, production of outputs, partnership strategy, formulation and performance.
The specific objectives of the outcome evaluation include:
  • Assess progress towards the outcome;
  • Assess the factors affecting to the outcome;
  • Assess key UNDP contributions (outputs), including those produced through "soft" assistance, to outcome;
  • Assess the partnership strategy.
Evaluation Questions
  • Relevance: Is the outcome still relevant to UNDP’s mandate, to national priorities and to beneficiaries’ needs?
  • Effectiveness: Is the outcome achieved or has progress been made towards it? Has UNDP made significant contributions in terms of strategic outputs;
  • Effficiency: what is the extent of UNDP’s contribution to the outcome versus that of its partners?
  • Degree of change: What were the positive or negative, intended or unintended, changes brought about by UNDP’s interventions?
  • Sustainability: Will the positive change in the development situation endure/continue in future?
 
Outcome Analysis
  • Are the stated outcomes, indicators and targets relevant for the context, needs and priorities of Ethiopia and those of UNDP?
  • What is the current status and prospects for achieving the outcome with the indicated resources and within the time frame?
  • Are the outcome indicators chosen relevant and sufficient to measure the outcomes?
  • How have other partners contributed to the achievement of the outcome and how instrumental has UNDP been in rallying this contribution.
  • How has UNDP influenced policy, institutions, cultural factors and gender in addressing the outcome?
 Output Analysis
  • What are the key outputs that have been or that will be most likely be produced by UNDP contribution to the outcome? Can causality linkages (results chain) be demonstrated between production of outputs and achievement of the outcome?
  • Are UNDP outputs relevant to the outcome?
  • Are the indicators appropriate to link outputs to the outcome
Output-Outcome Link
  • Can UNDP outputs or other interventions be credibly linked to the achievement of the outcome?
  • What are the key contributions that UNDP has made to the outcome
  • Given current interventions and in partnerships with other, actors and stakeholders will UNDP be able to achieve the outcome within the set time frame and resources?
  • In what way has UNDP been able to develop sustainable national capacity in this area
  • Has UNDP been able to respond to changing context and circumstances
  • What is the prospect for sustainability as a result of UNDP interventions related to the outcome?
  • Resources, Partnerships and Management Analysis
  • Were partners, stakeholders and/or beneficiaries involved in the design of the intervention relating to this outcome? If yes what was the nature of their involvement?
  • Are UNDP structures and working methods appropriate and supportive of the achievement of the outcome?
  • Does the project and institutional arrangements foster sustainability of benefits beyond the current UNDP support?
 Methodology
 
Flexible blueprints
 
Overall guidance on outcome evaluations methodologies is provided in the UNDP Handbook on Monitoring and Evaluation for Results and the UNDP Guidelines for Outcome Evaluations. Reference should be made to these documents when addressing issues of methodology.
 
Based on these guiding documents, and in consultation with UNDP Ethiopia, the evaluators should develop a suitable methodology for this specific outcome evaluation.

 

 

 

During the evaluation, the evaluators are expected to apply the following approaches for data collection and analysis:

  • Desk review of relevant documents (see attached list, documents and weblink);
  • Discussions with UNDP Ethiopia senior management and programme staff;
  • Interviews with partners and stakeholders;
  • Interviews with relevant programme staff;
  • Interview with partners;
  • Interviews with beneficiaries, etc.

There is no official blueprint for how to conduct an outcome evaluation. Each must be tailored to the nature of the individual outcome under review as well as the realities of time and data limitations. Two main limitations related to the outcome evaluation have been identified and these are the following:

  • Under the outcome “By end 2011, at least one EGC is identified based on the national strategic framework and ready to be operational”  there is only one project;
  • The outcome formulations have changed during the programme cycle 2077-2011;
  • Not all outcome indicators are in line to the outcome statement;
  • A new outcome was formulated in 2009, the project contributed to that outcome “Private Sector Development promoted as a motor of Economic Growth”, has nearly one year implementation.

Usually, an outcome evaluation begins with a review of change in the outcome itself, proceeds to an analysis of pertinent influencing factors, and then addresses the contribution of UNDP and its partners. It culminates in suggestions about how to improve the approach to results.

More in detail, the first step is for the evaluators to ascertain the status of the outcome. Evaluators should take the following steps: The role of an evaluator is to pass judgment based on his or her best professional opinion. Usually, an outcome evaluation begins with a review of change in the outcome itself, proceeds to an analysis of pertinent influencing factors, and then addresses the contribution of UNDP and its partners. It culminates in suggestions about how to improve the approach to results.

The first step is for the evaluators to ascertain the status of the outcome. Evaluators should take the following steps:

  1. Begin with the description of the intended outcome, the baseline for the outcome and the indicators and benchmarks used. Obtain information from the country office gathered through monitoring and reporting on the outcome. This will help inform evaluators of whether change has taken place or whether a deconstruction of the outcome statement has to be undertaken.
  2. Obtain contextual information beyond what UNDP has tracked. Before organizing an outcome evaluation, the country office will have undertaken preliminary data collection (contextual data as well as evaluations, monitoring reports, etc.) and, depending upon the country office, possibly started to analyze the data. An outcome evaluation is not only designed to tap UNDP-specific information about the outcome but also to derive contextual information from other sources that detail trends in policy formulation, changes in human development indices over time and other changes.
  3. Examine contextual information and baselines contained in project documents, the Country Programme Document (CPD) 2007-2011 and the Country Programme (for newer programmes), United Nations Development Assistance Framework (UNDAF) and other sources. These documents speak to the outcome itself, as opposed to what UNDP is doing about it, and how it was envisaged at certain points in time preceding UNDP’s interventions. The situation analysis section of both the Country Programme and the project document, for example, could each provide useful information and hyperlinks to other sources of information.
  4. Employ a means to validate information about the status of the outcome that is culled from contextual sources such as monitoring reports. To do this, evaluators may use interviews or questionnaires during the evaluation that seek key respondents’ perceptions on a number of issues, including their perception of whether an outcome has changed.
  5. Probe the pre-selected outcome indicators, go beyond these to explore other possible outcome indicators, and determine whether the indicators have actually been continuously tracked. The one or two indicators provided are unlikely to yield sufficiently relevant evidence of change in the outcome. In time, it is expected that the monitoring process will lead to modification in the outcome indicators themselves.
  6. Undertake a constructive critique of the outcome formulation itself (and the associated indicators). This is integral to the scope of outcome evaluation. Evaluators can and should make recommendations on how the outcome statement can be improved in terms of conceptual clarity, credibility of association with UNDP operations and prospects for gathering of evidence. 

This will follow with the activities, such as a series of meetings with the key actors in public, private at national, regional and local levels. In this regard, field visits will be conducted in order to undertake several interviews to focused groups and beneficiaries. Data and information to be collected should be evidence-based, as well as qualitative and quantitative in nature. Where there is required, evidence-based data need to be presented with digital pictures. As much as possible, the assessment will follow a participatory approach.

The international consultant will be provided with all the relevant documents for their review, prior to their arrival in Ethiopia. Once the consultants join in their assignments, their tasks, in consultation with the Team Leader and M&E officer, will focus on: (a) field visits and meeting with key stakeholders, (b) data collection and analysis; (c) development of a draft evaluation report; and (d) presentation of findings, recommendation and the way forwards.

What are the main factors (positive and negative) impacting the achievement of the outcomes.

Duties and Responsibilities

Deliverables

The consultants are expected to produce an Outcome Evaluation Report that highlights the findings, lessons learnt and recommendations. This report should follow the Outcome Evaluation Report Template and include all sections recommended therein (see attached template). The evaluators are also expected to produce an inception report before they commence on the bulk of the evaluation work. 

Once the draft evaluation report is developed by the consultants, it should be shared with all the key relevant stakeholders for their comments, views, suggestions and recommendations.
 
  • List of deliverables:
    • Inception report: (3 days after the commencement of the assignment): The consultants need to submit the overall methodological framework which should define: eventually the deconstruction of the outcome statement/indicator and how such would be assessed/evaluated, Data collection tools and instruments, data source, sampling approach, type of data analysis, reference indicators and benchmarks (if relevant)
    • First draft of the evaluation report: shared with M&E officer, Team Leader, stakeholders and comments incorporated where relevant.
    • Final Evaluation Report: High quality Evaluation Report with the implementable action plan /recommendations both in hard and softy copies. Moreover, disseminate findings among implementing partners. Dissemination kits (PowerPoint Presentation, reader-friendly summary version of report, etc.) 
    Timeframe
    It is expected that the evaluation would be undertaken starting from October 2011. The consultancy will be 22 days, 10 days home-based work, 10 days on the site, as per the following timeframe:
    • Main Activity
      Working Days Assigned
      Officers
      Transportation
      1

      Desk Review of the relevant documents

      4 (home-based)
       
       
      2

      Consultations with key stakeholders, including donors, at federal level

      4 +2 days travel

       
       
       

      Ethiopian chamber of commerce, WB, ECX
      1
       
       
       

      MOFED, MoI, MoT

      1
       
       
       

      Consultation with UNDP relevant staff , Oromia BOFED, Ethiopian investment office

      1
       
       
       

      Field visits to Eastern Industrial zones (Dukam, Fantalle)
      1
       
       
      4

      Data analysis and compiling, and drafting the Evaluation Report

      5
       
       
      5

      Arrangement for a one-day Stakeholder meeting. PPP
      1
       
       
      6

      Finalization and submission of Evaluation Report, Recommendation, Action Plan.

      5 (home-based)
       
       
       
      Total Days Required:
      22 days
       
       
       
      Team Composition and Required Competencies
       
      Two high caliber professionals, one national and one international, will be recruited as a Team Leader/ for this evaluation exercise. The international consultant (Team Leader) and the national consultant will be recruited for 22 days.
       
      International consultant:
       
      October 2011. The consultancy will be 22 working days, 10 days home-based work, 10 days on the site.
       
      Under the guidance of one international consultant and one national consultant/ M&E officer, the evaluation will be conducted in a timely manner. The two consultants will report to the Team Leader/ M&E officer. The consultants will work together as a team for achieving the deliverables of the evaluation with high standard that have to be accepted by UNDP.
       
      This evaluation requires senior professionals, these are: one Team Leader (international consultant), one national consultant, UNDP Team leader, UNDP M&E officer. The qualifications and experiences of the international consultant are, as follows:  
       
      Administrative and Logistic Support:
       
      The consultants will be recruited under the UNDP terms and conditions, applicable to the short-term IC contract holders, and undertake their assigned tasks and responsibilities under the direct supervision of the UNDP Team Leader, Governance Unit, and the M&E officer in collaboration with other UNDP Units, and other key stakeholders at federal, regional and local levels. During the evaluation period, the consultants will directly report to the UNDP Team Leader/ M&E officer.
       
      All the payments of the Consultants will be borne by the Programme. For the field trips and meetings, UNDP will provide logistic and administration support to the Consultants. In this regard, the Consultants are required to plan their trips in advance.
       
      Payments to Consultants: The national and international consultants shall receive their respective consultancy fees upon certification of the completed tasks satisfactorily, as per the following schedule:
       

        Installment of Payment/ Period

        Tasks to be completed by Consultants

        Payment to be made by UNDP

        1st Installment:

         

        Upon submission of the acceptable (1) draft Evaluation Report

        40 % of the total consultancy fees
        2rd Installment:
         

        Upon submission of the acceptable Final Evaluation Report
        60% of the total consultancy fees

    • Competencies

    Experience and Skills

    • At least ten years demonstrated experience in planning, implementation and management of the local development and /or economic development programmes at international level;
    • Extensive experience in assessment and evaluation of economic projects, at national and international levels;
    • Three to five years experience in providing high quality technical and advisory support to the governments at federal and local levels in local development policy, strategy and programme development;
    • Have excellent analytical skills and experience in developing high standard evaluation reports;
    • Working experience and familiarity with the development efforts of the international development partners, especially UN agencies and NGOs;
    • Sound understanding of the national and international development policies, strategies and programmes, and their implementation issues and challenges, especially in the area of development economics;
    • Exposure to or understanding of the Ethiopia’s socio-economic and geo-political contexts would be an asset;
    • Full proficiency in using computer software, such as Microsoft offices and internet.

    Language:

    Full proficiency in both spoken and written English.

    Required Skills and Experience

    Education

    Master’s Degree in Development Economic, or development related fields, with the participation in several international training courses, relating to local or/ and economic development, or capacity development.

    Documents to be submitted

    Interested individual consultants must submit the following documents/information to demonstrate their qualifications:

    1. Proposal: (not more than 450 words)
    (i) Explaining why they are the most suitable for the work

    (ii) Provide a brief methodology on how they will approach and conduct the work (if applicable)

    2. Financial proposal:
    3. Personal CV including past experience in similar projects and at least 3 references

    Payment Proposal

    Proposed payment for the consultancy work on lump sum rate in a separate sheet

    Evaluation

    Individual consultants will be evaluated based on the following methodologies:

    Cumulative analysis

    The award of the contract shall be made to the individual consultant whose offer has been evaluated and determined as:

    a) responsive/compliant/acceptable, and

    b) Having received the highest score out of a pre-determined set of weighted technical and financial criteria specific to the solicitation.

    * Technical Criteria weight; 70%

    * Financial Criteria weight; 30%

    Only candidates obtaining a minimum of 49 point and above would be considered for the Financial Evaluation

    Criteria
    Weight
    Max. Point
    Technical  (based on CV, Proposal and Interview)
     
     
    Minimum educational background and work experience (CV)
    10%
    10

    Understanding of scope of work and methodology (From Proposal)
    20%
    20

    Individual Competencies (Desk review or Interview)
    40%
    40
     
     
     
    Financial
    30%
    30