SCOPE OF WORK
The objective for this MTE is to examine the progress of Gov4Res against its original intentions, identify areas for improvement and given the changing governance context, identify new opportunities, recommend changes to update the project plan and approach.
The MTE will assess the following:
- Relevance, effectiveness, and efficiency of project
- Risks to sustainability
- Extent to which gender equality and social inclusion and human rights aspects have been considered
- Project structure
- Monitoring and evaluation approaches of the project
Evaluation Approach and Methodology
Gov4Res utilises a wide range of strategies and pathways to achieve change. It considers both technical and ‘political’ motivations and influences. It seeks to engage respectfully and effectively in different cultures and contexts, responding to the needs of the countries it works with. It deliberately works across various entry points in countries in order to maximise the likelihood of positive change and invests in systems (i.e., structures and processes) to ensure long-term adoption and sustainability. In several areas of work it seeks to influence others such as regional organisations and interactions between different national actors, working from behind and empowering others, rather than leading on all activities. It has a focus on working in agile and flexible ways, learning from experience and changing strategies as required.
While its original theory of change presents a concise summary of its core assumptions, in practice Gov4Res understands the change it is seeking to achieve is complex and that the connection between all these activities and outcomes is multifaceted, complex and dynamic. Consequently, it is anticipated that the MTE approach should be based in a critical epistemology, able to engage effectively with a non-linear systems-based project approach.
Gov4Res recognises that its various stakeholders have overlapping and different views about what changes are important. The donor partners have clear but different strategies and outcomes they wish to achieve. Pacific government partners, Pacific people and communities and regional organisations all have their views on the prioritisation and significance of the changes that might be achieved through the project. The MTE needs to be gender-sensitive and socially inclusive, able to accommodate and give attention to assessment from these various different worldviews.
The MTE approach will accommodate and identify differences in assessment, values, and understanding of impact for stakeholders, and provide methodological approaches that create dialogue and exchange between stakeholders and their different perspectives. The approach should be sensitive to Pacific Island approaches, and respectful of the knowledge of Pacific Islanders.
Specific data collection, analysis and engagement techniques will be agreed as part of the evaluation plan prior to commencement of the MTE. However, it is anticipated that the evaluation team will demonstrate considerable skill in analysis and sense making that is inclusive of project stakeholders and provides opportunities for women, marginalised groups and Pacific country stakeholders to engage with and assist in data analysis and recommendation development. The methodologies proposed by the evaluation team should also support and facilitate active dialogue between stakeholders and their different perspectives.
Gov4Res has an established Monitoring, Evaluation and Learning Framework (MELF) based on its original project theory of change. It has a comprehensive data management system which includes evidence against outcomes and outputs and the project reports regularly against its outputs and outcomes to its various donors and oversight Board. As noted, the original theory of change has clearly defined pathways and strategies for change and evidence has been collected against these pathways. In preparation for this evaluation the project has undertaken an extensive audit of existing information and evidence in each of its country locations and its regional work. These country and regional briefs, annotated against the existing evidence, will be provided to the MTE team in order to assist the team to efficiently understand the scope and variety of work by the project to date. While some verification of these briefs will be required, this extensive preparation will enable the MTE team to focus its data collection and analysis on areas which are outside of the present project MELF, identifying new information and insight for the project stakeholders.
It is expected that the MTE will make use of this existing evidence base and develop additional methodologies for data collection, analysis and examination that complement rather than duplicate the existing information. The team will be expected to have extensive expertise in qualitative and quantitative methodologies including, as indicated, the capacity to use data collection, analysis and engagement techniques that are appropriate to different stakeholders.
The team should be familiar with Pacific approaches to evaluation. Ideally the evaluation team will bring expertise in feminist and/or indigenous methodologies and/or other methodologies drawn from critical evaluation approaches, alongside experience in traditional methodologies (ie. interviews, surveys, observation, focus groups etc).
DETAILED SCOPE OF MTE
The MTE team will assess the following categories of project progress:
The MTE will assess the ongoing relevance of Gov4Res, given the changing context since project commencement. This will require examination of the initial project analysis and strategy development, how adequately this has been updated in response to changing context, wider examination of key contextual influences (both enabling and disabling), and how adequately the project has responded to or is positioning to respond to these conditions.
The relevance of Gov4Res should be considered from the perspective of different stakeholders including partner governments and Pacific Island communities, civil society organisations, and private sector. The projects coherence with other interventions, especially those of the donor partners, regional organisations and UNDP should also be reviewed.
The MTE will recommend options to support ongoing project relevance and coherence, giving due attention to these different perspectives. To assess relevance and coherence, the following should be considered (but should not limit the evaluation):
- How well does the project and its outcomes align with the priorities of local government and local communities in the focal PICs?
- How well does the project and its outcomes align with PIC’s National Government development priorities and with regional development priorities?
- How well does the project align with national and regional gender equality and other social protection commitments?
- Does the project objective fit UNDP Pacific strategic priorities?
- How well does the project align with similar interventions in the region, especially those supported by its donor partners?
- In what ways has the project responded and adapted to maintain relevance and coherence for all stakeholders?
The MTE will verify project effectiveness utilising available information (see discussion around methodology below), together with additional evidence collected as required. The MTE will consider in particular, how effectively the Gov4Res project has progressed against its original outcomes and outputs as outlined in the original project theory of change. As required, the MTE will examine core assumptions under the original theory of change and test how well these have held throughout project implementation to date. The MTE will recommend options for further development and maturing of the project theory of change that will support increased project effectiveness.
To assess effectiveness, the following should be considered (but should not limit the evaluation):
- What have been the key results and changes achieved by the project to date?
- To what extent will the project meet its original outcomes within the current program phase? Do these remain practical and feasible?
- Do the project assumptions and project theory of change continue to address the key factors which are likely to enable or challenge the progress of this project?
- Has the project been able to respond effectively to new emerging opportunities?
- In what ways should the project theory of change be further developed, given progress to date and changes in project context?
- What implications do recommended changes to the project theory of change have for project strategies, monitoring and evaluation, and reporting?
Gov4Res operates through a range of strategies and pathways to achieve change. The MTE is an opportunity to review the efficiency of the major project strategies. That is, given the resources available, which of these strategies most efficiently contributes to project implementation? In particular the MTE will examine the value being achieved from the following strategies:
- “From within” approach of embedding focal points within government
- Agile/adaptive programming
- Demonstration of risk-informed development through community infrastructure and development program
- Regional policy support and research to achieve scale
- Portfolio approach of interventions, that integrate across difference governance levels
- The MTE will recommend options to further develop the current project strategies and/or expand or change strategies, in order to support efficient progress towards project outcomes.
- To assess efficiency, the following should also be considered (but should not limit the evaluation):
- Has the project been efficient in leveraging resources and partnerships that are currently contributing to, or have contributed to achieving outcomes?
- In what way have changes in the context affected project cost effectiveness?
- What changes ought to be made in project strategies in order to ensure the most efficient approaches to project implementation?
Gov4Res works with and through PIC governance systems and practices to promote sustainability of the reform agenda. The MTE will assess the extent of take up of project activities by in-country systems. It will recommend options and areas of work where the project should expand or undertake further activity to support sustained PIC led outcomes.
Considering the progress of work in this current phase and the likely options for future phases of the program, the MTE will examine how the project can most effectively support sustained Pacific Islands-led action for risk-informed development.
The MTE will also identify areas for further research and enquiry is required in order to develop additional activities and strategies that will support sustained outcomes beyond the life of this project. In considering sustainability of outcomes the MTE will give particular attention to the principle of localisation.
To assess sustainability, the following should be considered (but should not limit the evaluation):
- How effectively has the project worked through PIC governments’ systems and practices to introduce reform measures?
- In what ways has the project partnered with key actors on the ground (including communities and local government) to ensure program benefits are sustained?
- What further development of work areas is required to increase the sustainability of project outcomes?
- In what ways does this project support the core principles of localisation in the Pacific? In what ways could this be further improved?
- Human Rights
line with UNDP principles the MTE should assess to what extent human rights considerations are included in the project design and implementation
- To what extent does the project adhere to and further supports human rights principles?
- To what extent does the project integrate or consider human rights-based approaches in the design and implementation of the project?
- Gender Equality and Social Inclusion
Gov4Res proposes that it is impossible to risk-inform development without understanding and addressing the underlying vulnerabilities that arise due to structural inequalities that prevent women and marginalised groups from contributing to and benefitting from that development. To ensure that the process is equitable, and benefits reach marginalised groups, the development process must be informed by diverse voices.
The project has recently developed a Gender Action Plan. The MTE is timely as the project will use the outcomes of the review to further refine its strategies for implementing its Gender Action Plan and refine its GESI indicators for the project
The MTE will assess the quality and value of the Gov4Res gender equality and social inclusion (GESI) strategies, as outlined in its GESI Action Plan, including how comprehensively and effectively the project has partnered with women, marginalised groups, including people living with a disability, and those marginalised by other intersecting social identities (e.g., age, gender, ethnicity, living in rural or remote areas, etc.), in project activity planning, implementation and assessment.
The MTE will provide detailed recommendations for further improvement in the Gov4Res strategies for GESI, including opportunities to engage with other organisations and practitioners working on these issues (e.g., Ministry of Women, UN Women)
To assess the integration of GESI into Gov4Res, the following should be considered (but should not limit the evaluation):
- How has the project contributed to gender equality, particularly in terms of women's empowerment?
- How has the project contributed to equality and empowerment for other marginalised groups (e.g., people living with a disability, or people marginalised by other intersecting social identities (e.g. age, gender, ethnicity, geography etc.) through project activity planning, implementation and assessment.
- How is this participation of men, women and vulnerable groups contributing towards achievement of the project outcomes?
- How effectively have the measures or processes as outlined in the GESI Action Plan integrated GESI into project?
- How could the project further improve and assess its strategies for gender equality and social inclusion?
- What additional strategic partnerships should be cultivated to advance GESI in risk-informed development?
- Project Structure
Gov4Res is currently designed as a project under the UNDP Resilience and Sustainable Development Unit. It receives funding from several donors and allocates those funds to different areas of project activity and in some cases, different locations. Going forward, the project seeks to make the most efficient use of donor partner funds, as well as provide the maximum accountability for those funds.
The MTE will explore options for the project structure going forward, considering likely future phases of the project, and identify options for how the project can be most efficiently structured to meet UNDP processes and respond to donor partner accountability and reporting requirements.
To assess the appropriateness of the project structure, the following should be considered (but should not limit the evaluation):
- How should the project be structured to meet UNDP processes, respond to donor partner accountability and reporting requirements and meet its intended outcomes?
- Does the team have the required skills and experience, or technical partnerships in place to deliver the outcomes of the project?
- Are there additional activities, relevant to project stakeholders and in line with project outcomes, which could be included in future development of this project?\
- Monitoring, Evaluation and Learning
To assess the appropriateness of the project monitoring and evaluation, the following should be considered (but should not limit the evaluation):
- How comprehensively has the project collected, analysed and reported verifiable information about its progress?
- Are there missing indicators that are cost-effective and more impactful to measure?
- In what way could the project Monitoring, Evaluation and Learning Framework be further developed and improved to ensure accountability to all stakeholders and support further project improvement?
- How is the projects’ learning being captured and shared, and are there ways to improve information capture and its communication to various audiences?
Duration of the Work
This assignment is anticipated to take place between 1 June 2022 – 31 January 2023. The assignment is home-based, and payments are output based. Travel is required to Fiji and may be undertaken to other one or two Pacific locations if COVID restrictions allow. The expected level of effort for the MTR consultant(s) is approximately 80 days in total.
Country and Regional briefs
Project briefs outlining current activities, theory of change and achievements, annotated with available evidence, prepared for each project country and for the project regional activities.
Gov4Res M&E team
Preparation of evaluation plan Inception Report
Draft Inception report in line with UNDP evaluation norms and standards, following initial discussions with project team and other relevant stakeholders.
Review of draft evaluation plan Inception Report
Comments on the draft Inception Report, provided by the Evaluation Reference Group and UNDP, consolidated by the evaluation manager.
Incorporation of comments
Revised Inception Report drafted
Final Inception report submitted
Fiji and the other countries if COVID restrictions permit
Data collection, on-site analysis.
Homebased or Fiji
Initial findings presentation to evaluation reference group and other stakeholders as required. Presentation submitted.
Draft evaluation report submitted
Review for quality assurance and scope. Identify factual errors and clarity and comprehension
Comments on the draft evaluation report, provided by the Evaluation Reference Group and UNDP, consolidated by the evaluation manager.
Consideration of comments
Revised draft evaluation report
Final review by UNDP IRMU
Revised draft evaluation report submitted to UNDP IRMU; draft Evaluation Brief submitted
Incorporation of comments and finalisation of report and Evaluation Brief
Revised draft evaluation report, with comments from UNDP IRMU consolidated
Final evaluation report; Evaluation Brief and presentation of evaluation results.
Audit Trail Form
Project Management: Finalise Evaluation Follow-up Plan
Final Evaluation Follow-up Plan to be cleared by IRMU
Project Management: Disseminate final evaluation report
Final evaluation report disseminated to internal and external stakeholders
The MTE will include travel to Fiji where Gov4Res has its main base. If COVID restrictions allow, travel will be expected to other locations (a minimum of 2 other Pacific countries). If further travel is not possible then the evaluation team will be required to undertake remote data collection and analysis. The team will be expected to manage this remote process in ways which ensure the methodological standards outlined above are maintained.
Limitations and Risks
With restrictions due to COVID-19, and in order to stay within reasonable time and budget limits, it will not be possible for the evaluation team to travel to all project locations nor consult all project stakeholders. As a result, there is a risk that those stakeholders who are more difficult to reach either because of location or other factors, will have limited opportunity to contribute to the evaluation. This is a limitation which will need to be carefully managed in the MTE planning and it is expected that the evaluation team will give due consideration to this risk in the comprehensiveness of their data collection.
There are limits to the expertise that an external evaluation team is able to bring. For a project such as Gov4Res with multiple technical and other work areas, there is some risk that the evaluation team will not be able to fully understand or appreciate all of the work undertaken by the project especially given that strategies and activities are localised to particular countries and sectors. Gov4Res will aim to support the evaluation team as far as possible to access a full understanding of the project activities and achievements, providing country and regional briefs, support in technical and other inquiries and directing the team to additional areas or people who can provide the required information. However, it is acknowledged that there is a risk that the evaluation team will gain a partial or incomplete perspective of the entire project operations. To mitigate this risk the Evaluation Reference Group will be tasked to review the evaluation team findings and analysis and recommend where further data collection might be required. The evaluation team will also be expected to use multiple data collection and analysis methods to test findings and interpretations with different stakeholders as they proceed.
A time bound evaluation is limited in the range of areas and issues it is able to examine in-depth. While the terms of reference for this evaluation have been deliberately limited to those areas of current significance for the project, it is also recognized that each of the MTE objectives will likely involve considerable methodological enquiry. In order manage this it will be important for the evaluation team to maintain communication with the Evaluation Reference Group and with other stakeholders including donor partners, to ensure expectations are realistic and are being satisfactorily addressed.
This evaluation will be conducted in accordance with the principles outlined in the UNEG ‘Ethical Guidelines for Evaluation’. The consultants must safeguard the rights and confidentiality of information providers, interviewees and stakeholders through measures to ensure compliance with legal and other relevant codes governing collection of data and reporting on data. The contractor must also ensure security of collected information before and after the evaluation and protocols to ensure anonymity and confidentiality of sources of information where that is expected. The information knowledge and data gathered in the evaluation process must also be solely used for the evaluation and not for other uses without the express authorization of UNDP and partners.” Contractor will be held to the highest ethical standards and are required to sign a Code of Conduct upon acceptance of the assignment.