Historique

1.Purpose of the Working Paper

The purpose of this working paper is to explore existing approaches for measuring impact in Gender Equality and Empowerment of Women (GEEW) both within UN Women and other organizations. It will outline the conceptual background for impact evaluation and analyze experiences with impact evaluation of GEEW in UN Women and other organizations. The paper will also assess the UN Women institutional context and necessary conditions for measuring impact in the organization. Finally, the paper will provide recommendations on possible approaches for measuring the impact of UN Women’s work and how UN Women can enhance its efforts in this area. The purpose of the paper is also to contribute to global knowledge and national evaluation capacity for evaluating gender equality and empowerment of women. Therefore, the target audience is both UN Women staff and senior management and global community of evaluators or programme managers.

2.Purpose and Definition of Impact Evaluation

As established by UNEG (2013) Impact Evaluation is ideally embedded within broader monitoring and evaluation systems.  The key questions to which impact evaluation can provide invaluable and perhaps unique answers include the following:

• Did the intervention make a difference?
• What specific contribution did the intervention make? (Alternatively: “What specific part of this difference can be attributed to the intervention?”)
• How was the difference made?
• Can the intervention be expected to produce similar results elsewhere?

These questions cover a broad range of issues from accountability (including value for money) to lesson learning (for replication and scaling up of the effects of the intervention).

According to UNEG most UN Agencies including UN Women have adopted the DAC definition of impact “Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended” with some adaptations to account for specifics of their key target groups.

The DAC definition has several important elements: Impact is about “effects produced by a development intervention”. It is therefore about “cause and effect” and thus specifically addresses the issue of attribution, which incorporates the concept of contribution. It should be noted that attribution-based definitions of impact do not require that effects be produced solely or totally by the intervention. They anticipate the co-existence of other causes so that the intervention will have contributed to the demonstrated effects. The DAC impact definition specifically includes the possibility of partial attribution, or contribution, through its inclusion of secondary and indirect effects.

Another important aspect of the DAC definition of impact is that it focuses on “long term effects”. According to the DAC Glossary, outcomes are the “likely or achieved short-term and medium-term effects of an intervention’s outputs”. The DAC definition therefore draws attention to a longer time scale, in which short and medium-term effects (outcomes) have played some part in the generation of “long-term effects” (impacts). It should be noted that the concept of a “long-term effect” does not define when in the overall results chain such an effect can begin but highlights its duration.

3.UN Women’s understanding of changes at impact / goal level

UN-Women’s global 2018- 21 Strategic Plan is based on the goal / impact statement “Achieve gender equality and empower all women and girls” and further explained through the following Theory of Change (ToC):

UN-Women Goal ToC:

IF a comprehensive and dynamic set of global norms, policies and standards on gender equality and the empowerment of all women and girls is strengthened and implemented, IF women lead, participate in and benefit equally from governance systems, IF women have income security, decent work and economic autonomy, IF all women and girls live a life free from all forms of violence, IF women and girls contribute to and have greater influence in building sustainable peace and resilience, and benefit equally from crisis prevention and humanitarian action, and IF the entire UN system delivers on its commitments to gender equality and women’s empowerment; THEN women and girls will be able to exercise their rights on an equal footing with men and boys and equally contribute to and benefit from development

The UN WOMEN global ToC provides the basis for the Strategic Plan (SP) development results. It defines necessary conditions and actions, how each Outcome of the SP will contribute to transformative change for women and girls, and how UN WOMEN will address these through its composite mandate of normative, coordination and operational activities. The UN WOMEN global 2018- 21 SP also refers to Flagship Programme Initiatives (FPIs) as ‘partnership vehicles to coalesce partners around common goals’. FPI Theories of Change represent a proposed set of activities that must be undertaken by all partners to deliver transformative results for women and girls.

In other words, ‘transformative changes’ are pitched at goal/ impact level and achieved through collaborative efforts by UN WOMEN and its partners.

The UN WOMEN 2018- 21 Strategic Plan includes an Integrated Results and Resources Framework (IRRF) with Impact, Outcome and Output level indicators defined as follows:

IRRF Impact indicators (total = 5): These indicators measure the highest-level result that the organization aims to achieve in the long-term, with a vision to 2030. Impact-level results refer to ambitious changes in human lives, particularly the lives of women and girls. These results require action from many actors, including UN-Women, over a long period of time. All of UN WOMEN’s impact-level indicators reflect international published data sources, such as the annual Secretary General’s report Progress towards the Sustainable Development Goals

IRRF Outcome indicators (total = 27): Outcome-level results reflect key results to which UN WOMEN contributes in order to effect change at the impact level. They measure phenomena that often do not significantly change on an annual basis. 70% of all outcome indicators in the UN WOMEN Strategic Plan are either SDG indicators or rely on internationally published data sources, such as Demographic Health Surveys, World Bank databases, or official reports of the Secretary-General.

IRRF Output indicators (total = 51): Output-level results are the near-term, direct consequence of UN WOMEN interventions. Each output is designed to translate global norms into results for women and girls, in collaboration with the UN system and its partners, based on a common theory of change, which supports a division of responsibilities and is adjusted to country contexts and capacities. Output-level indicators have been designed in a way that they can be fully attributable to UN WOMEN and capture the direct contribution of the Entity.

4.Analytical Frameworks for Gender Equality and Empowerment of Women (GEEW)

The previous section has clarified concepts on Impact Evaluation and the scope and type of expected changes at different levels of the results chain. Now it is equally important to come towards a common understanding on what constitutes GEEW. In other words, the definition and analytical framework for GEEW has important implications on how GEEW can (or not) be measured in the context of an impact evaluation. A quick literature review already indicates that there are many different definitions of empowerment and consequently the analytical frameworks for measuring women’s empowerment diverge. As brief example, the UN WOMEN Country Portfolio Evaluation (CPE) Guidance recommends the Women’s Empowerment Framework developed by Sara Longwe[1] as a way to conceptualize the process of empowerment. It includes 5 dimensions of analysis: Welfare (improvement in socioeconomic status), Access (increased access to resources), Conscientisation (recognition of structural forces that disadvantage women), Mobilization (implementing actions related to the conscientisation of women) and Control (control of resources). OXFAM (2017) has developed a Women’s Empowerment Framework which recognizes 3 levels at which change can take place: personal (how a woman sees herself), relational (power relations in the household and community) and environmental (social norms and attitudes, political and legislative framework). The Abdul Latif Jameel Poverty Action Lab (J-PAL) suggests using the 3 interrelated dimensions of resources (material, human, social), agency (participation, voice and influence) and achievements (meaningful improvements in well-being and life outcomes) for measuring empowerment.

These examples illustrate that there is some degree of overlap between the different GEEW frameworks. However, they also demonstrate how the very definition of GEEW has consequences on potential indicators for measurement, data collection sources, evaluation questions and overall evaluation design.

5.Evaluability Assessment of the UN WOMEN 2018-2021 Strategic Plan

In 2018 UN WOMEN undertook an Evaluability Assessment of its 2018-2021 Strategic Plan which highlighted important findings and conclusions regarding the current UN WOMEN institutional context and its ‘readiness’ for assessing results, including GEEW results at impact level. They are summarized below:

Finding 1: UN Women’s Theory of Change is well articulated and logical with an identification of assumptions and risks. However, UN Women’s continuous causal results chain and the Strategic Plan’s Theory of Change do not link easily and logically to the country level M&E Framework.

Finding 2: UN Women has articulated an impact statement for its results framework at a very high level which reads more as a vision statement, and many outcome statements read more like goal/ impact statements in UNDG accepted RBM terminology.

Finding 3: UN Women’s RBM system has been improved over the years. However, current IRRF indicators are appropriate for accountability and outreach purpose; they are less well developed to capture and explain change and manage for results. For example, indicators are largely quantitative leaving little space for capturing and understanding “how” or “why” change may or may not have happened, which limits its potential as a tool for learning.

Finding 4: The UN Women Theory of Change includes multiple elements that will contribute to GEEW. However, because it is a generic TOC it does not capture the context-specific nature of gender inequality and nor the multi-directional pathway that is often the reality. The IRRF is less useful to understanding UN Women’s contribution to transformational change.

Finding 5: “No one left behind” principles should be clearer addressed and integrated; current UN Women’s IRRF does not capture the multiple dimensions of exclusion across and within outputs and outcomes.

Finding 6: The UN Women IRRF does not adequately capture all impact level results of UN Women in the three mandate areas.

Finding 7: Within the design of the SP and the IRRF, UN Women coordination role is primarily captured as an institutional output in the OEEF but not well reflected on development results and impact level.

Conclusion 1: To fully evaluate the Strategic Plan, a comprehensive results architecture and reporting must be in place.

Conclusion 2: The multi-purpose nature of the SP should be recognized as the internal logic is not always clearly articulated and evaluable.

Conclusion 3: UN Women needs to capture LNOB in the IRRF more sufficiently; obtaining the disaggregated reporting necessary is challenging and methodologically complex, as social exclusion is very context specific.

Conclusion 4: As the impact statement is set at a very ambitious level, UN Women needs to have a common understanding of impact at global, regional and country level.

Conclusion 5: Recognizing the methodological complexity, UN Women needs to develop better methods to track the coordination mandate’s contribution to development results.

In addition, it is important to consider the UN WOMEN triple integrated mandate in terms of (1) Supporting the formulation of global and national gender policies, norms and standards, (2) Helping countries implement these policies through technical and financial support and (3) Leading and coordinating the UN system’s work on gender equality. The various components of the UN WOMEN mandate are inter-related and in most cases change does not occur in a linear, unidirectional way but through iterations, feedback loops and with high degrees of unpredictability. The complex and long-term trajectory of much GEEW work and the indirect way in which impacts further down the results chain may come about poses additional challenges for evaluating longer-term impacts of GEEW.

[1] For details on the Sara Longwe framework see http://awidme.pbworks.com/w/page/36322701/Women%27s%20Empowerment%20Framework#_ftn1

Devoirs et responsabilités

1.    Methodology

The working paper will be developed by an international consultant under supervision of the UN WOMEN Independent Evaluation and Audit Service (IEAS). The IEAS staff will manage the consultant and support the process. A Reference Group will be constituted to provide feedback and quality assurance during the process.

Some of the questions that will be explored are:

1) How to measure the impact/contribution of UN Women coordination and normative work?

2) How to measure UN Women’s contribution to development outcomes?

3) How to measure UN Women’s contribution to SDGs?

The methods of data collection will consist of the following:

• Desk review of relevant literature and background documents
• Mapping of UN WOMEN indicators in Global 2018-21 Strategic Plan (SP) and in Country Strategic Notes (SNs) to understand how UN WOMEN is measuring GEEW results
• Face to face/ Skype interviews with key UN WOMEN staff from IES, Senior Management and M&E staff at HQ, Regional and Country level; and potentially with other international organizations with substantive experiences in measuring impact
• A survey to UN WOMEN staff to understand interest and experiences, opportunities and challenges with regard to measuring long-term GEEW results

The consultancy with have a duration of 41 working days. It will be home-based and include travel to UN Women HQ in New York as well as [tbd] country visits.

The working paper will be based on the following draft outline:

1) Purpose and Definition of Impact Evaluation

2) Conceptual frameworks for measuring GEEW

3) Impact Evaluations of GEEW

• Impact Evaluation experiences in UN Women (Safe Cities Initiative, other examples e.g. Fiji)
• Impact Evaluations of GEEW: Experiences by other organizations (UN, INGOs, Research Centers etc.)
• Emerging issues, opportunities and challenges in Impact Evaluations of GEEW

4) Necessary conditions for undertaking quality Impact Evaluations in UN Women

• Programme design, Theory of Change and indicators
• Instruments for quantitative and qualitative M&E data collection
• Potential Impact Evaluation Design and Methods
• Evaluation governance, roles and responsibilities
• Mechanisms for ethical conduct of Impact Evaluation
• Quality control for Impact Evaluation
• Impact Evaluation budgeting

5) Recommendations

• What systems or support (including financial) does UN Women require to undertake Impact Evaluations (IEs)?
• Looking at the UN Women integrated mandate, are there specific areas of work that lend themselves to impact evaluation?
• When should IEs be done, where and how?
• Should IEs be piloted or institutionalized in UN Women?
• If UN Women should not undertake IEs, which type of UN Women evaluations could provide the necessary evidence?
• How can UN Women disseminate findings of this paper while also contributing to national evaluation capacity building efforts?

2. Deliverables

The consultancy is expected to commence in April 2019. The activities, estimated working days and deliverables are as follows:

3. Communication strategy

The IEAS in consultation with the Reference Group will develop a plan for communicating and disseminating the findings of the working paper using e.g. online dissemination and a webinar. Different products may be developed for internal users and external partners. The different channels for communication and sharing information will be explored in consultation with the consultant.

Compétences

 Functional Competencies:

• Advanced evaluation expertise and experience in a wide range of evaluation approaches including impact, gender and human rights responsive, and mixed methods.
• Previous experience in conducting corporate and complex multi-stakeholder evaluations, preferably for the United Nations system.
• Knowledge of the relevant international frameworks on gender equality and women’s empowerment, gender mainstreaming, and gender analysis; experience/knowledge of women’s movements.
• Strong experience and knowledge in human rights, the human rights-based approach to programming, human rights analysis and related UN mandates.
• Demonstrable analytical, communication and drafting writing skills in English.
• Fluency in English.

 Corporate Competencies:

• Demonstrates integrity by modelling UN values and ethical standards, including UNEG norms;
• Acts as a team player and exercises diplomacy, tact and politeness; and
• Displays cultural, gender, religion, race, nationality and age sensitivity and adaptability.

Qualifications et expériences requises

Education:

Post graduate degree in a field of relevance for the assignment: Gender, Sociology, Political Science, Anthropology, or other Social Science degree.

Experience:

The consultant must have at least 15 years of evaluation experience with minimum 10 years’ experience in designing, managing and undertaking impact evaluations and applying human rights and gender equality-based approaches to evaluation. Must have demonstrated experience implementing both quantitative and qualitative data collection methods and triangulation of data. Must have specific experience in the field of Gender Equality and Women’s Empowerment.