Evaluating what works for the people we serve
2022 in review:
2022 was marked by significant instability. Many crises have unfolded in the past year, hitting vulnerable communities the hardest, including in Ukraine, Afghanistan, Ethiopia and – at the time this report is being written – southern Turkey and northern Syria.
In light of these unprecedented times, the role of evaluation as a critical tool to inform evidence-based decision-making is more important than ever. Through evaluation, we can see what works and what doesn’t, shine a light on good practice, and take corrective action where needed.
Despite the challenges of the last year, the Evaluation Office at UNHCR remained committed to its vision of supporting UNHCR with evidence, data, and learning for better results for the people we serve. Evaluation is a critical part of any results-based management (RBM) system, providing accountability and strategic analysis to inform agile management and decision-making.
The new Evaluation Policy, approved by the High Commissioner in October 2022, creates an enabling environment for an organization-wide approach to boost the use of evaluation as a routine management tool at all levels. One of the main changes in the policy is the establishment of coverage targets for different types of evaluations. The new policy introduces a requirement for management to report on the implementation of accepted recommendations starting from 2023.
As we move into 2023, we would like to take the time to share with you the progress we have made last year towards achieving our strategic goals under the four pillars of our work:
OVERVIEW OF EVALUATIONS COMPLETED IN 2022
Evaluations bring value when they inform thinking and programming, provide assurance of alignment with the organization’s strategic directions, as well as generate new knowledge and evidence that, in some cases, may be contrary to commonly held views. Real change can sometimes only be visible several years after the evaluation has been completed.
Some examples of how evaluations are having impact include:
The Evaluation Office continued to collaborate with operations, regional bureaux and headquarters divisions to support the monitoring and evaluation component of multi-year strategies which form an integral part of COMPASS, the UNHCR results-based management system for planning and budgeting. This was undertaken in several ways, including by supporting multi-functional teams in the development of regional and country-level monitoring and evaluation plans; contributing to COMPASS guidance and administrative instructions; providing resources for online training; revising the online course on multi-year monitoring and evaluation plans; and co-delivering virtual sessions to over 200 members of the monitoring and evaluation community of practice.
UNHCR's Repatriation Programmes and Activities
The 2022 Evaluation of UNHCR's Repatriation Programmes and Activities found that reflecting the voice and aspirations of refugees and better capturing the complexity of their decision-making on return remains key in both the design and implementation of UNHCR’s voluntary repatriation activities.
UNHCR’s operating models and guidance in support of voluntary repatriation are outdated and lack adaptiveness to complex environments. UNHCR’s effectiveness in the coordination of repatriation operations is widely recognized. Long-term reintegration support, however, still poses significant challenges. Challenges remain in UNHCR’s consistent and systematic collection and sharing of operations-level data on repatriation and reintegration activities. Area-based approaches offer promising avenues for reintegration programming, as they can facilitate multi-stakeholder engagement and increase the sustainability of returns.
Informed by the evaluation, UNHCR has updated its guidance documents on repatriation, including the Handbook for Repatriation and Reintegration Activities (2004) and the Voluntary Repatriation handbook for International protection (1996).
The Evaluation of UNHCR’s support to strengthen national asylum systems found that UNHCR has made good progress over the last few years, with its work on asylum capacity development becoming more comprehensive and developmental, in recognition that supporting national systems development is inherently multidimensional and complex – political, as well as technical, institutional as well as individual – and involving multiple actors, requiring sustained support over a long period of time.
UNHCR, though, still has some way to go. In some situations, the assumption has been made that substituting or adding capacity, for example, embedding contract staff in Government agencies has a capacity development effect or outcome.
The strength of UNHCR’s advocacy for asylum capacity development greatly increases when part of an intergovernmental process. The evaluation team found that the organization is good at using political openings but less adept at building or joining networks or partnerships that can exert more influence on asylum capacity development than by us acting alone.
UNHCR accepted all the recommendations and developed a full set of actions in the management response approved in November 2022 to be carried out over the next two years, including consolidating a capacity development approach, increasing engagement with partners, ensuring capacitated staff, managing risks and increasing the evidence base.
I want to convey our appreciation for the very consultative process and the excellent recommendations that came out of the Asylum Capacity Development evaluation. Very relevant recommendations on a more strategic and holistic view to asylum looking at root causes of backlogs including galvanizing opportunities to work with other actors in advocating for legal pathways for admission for those not needing international protection which in turn alleviates the asylum system for those who really needed it.
Evaluating UNHCR’s response to COVID-19 has been critical to understanding how the organization mitigated the negative impacts of the pandemic on displaced and stateless persons. Three evaluative exercises have been undertaken: a synthesis of evidence on UNHCR’s COVID-19 response drawn from 27 UNHCR evaluations over the period 2020-22; and two joint evaluations – one led by UNHCR, the Governments of Colombia, Finland and Uganda, the Active Learning Network for Accountability and Performance, and the COVID-19 Global Evaluation Coalition focused on the protection of the rights of refugees during the COVID-19 pandemic; and the other, an inter-agency humanitarian evaluation of the COVID-Global Humanitarian Response Plan, where UNHCR engaged as part of the management group.
These evaluations revealed how protection actors made extraordinary efforts in support of refugee rights in the face of an unprecedented global health emergency. Despite many challenges, the humanitarian community provided a safety net for many millions of people who otherwise would have likely gone without assistance. Furthermore, working under difficult circumstances, UNHCR and other humanitarian agencies, particularly national and local NGOs, showed remarkable adaptability, courage and tenacity in delivering a coordinated response at an unprecedented scale. There was consensus that refugees and refugee-led organizations played a key part in the response.
Despite these efforts, and sustained contact with persons UNHCR serves, under the pressures of remote delivery, not all the needs of vulnerable groups were met. Registration and refugee status determination were challenging to carry out, while resettlement met with, not surprisingly, major barriers. The remote approach to delivery placed those with specific needs at greater risk. Delicate or sensitive issues such as trauma, mental health issues, and sexual and gender-based violence were difficult to address remotely, and it was also harder to assess participants’ well-being via video or telephone calls. However, in many other areas, the remote approach stimulated innovation, allowing the maintenance of many services which previously relied on face-to-face contact. It has also created new delivery modalities that could be used in future emergencies (e.g., child protection and gender-based violence case management and mental health and psychosocial support, telehealth for health responses).
Overall, the commitment by UNHCR to ‘Stay and Deliver’ to those in need was identified to be integral to the organizational DNA, recognized by partners and valued by the people UNHCR serves. The double burden was for staff, challenging, navigating operational delivery amid pandemic conditions, but the lessons from the evaluations are being taken by the organization as it refines and adapts its approaches.
UNHCR’s Response to Multiple Emergencies in the Central Sahel Region: Burkina Faso, Niger, Mali
Faced with unprecedented challenges, UNHCR was largely effective in responding to the needs of diverse populations in accessible areas of the Central Sahel Region. Stakeholders positively viewed the scale-up of the response following the emergency declaration, corresponding to an increased budget, protection assistance capacity, and partner coordination. However, UNHCR’s response faced several major challenges, including insufficient resources for the scale of the response needed, the insecurity and volatility of the Sahel context, and the COVID-19 pandemic, which led UNHCR to declare a global L2 emergency one month after the Sahel declaration.
Significant differences in coverage emerged across countries per population group. There was a lack of consistency in defining UNHCR’s responsibilities toward internally displaced people (IDPs). The evaluation found that the IDP Policy and the Emergency Policy lack concrete guidance on how to achieve goals or mechanisms to evaluate effectiveness, offering broad objectives rather than specific directives.
The evaluation recommends the Sahel operations to be guided by a multi-year regional strategy considering the scale and trend of the Sahel crisis. The Sahel operations must be supported by systems and tools that facilitate and promote the roll-out, adoption, and implementation of emergency and IDP policies. The evaluation further recommends that emergency declarations must be accompanied by a strengthening of information systems, monitoring and evaluation, and knowledge management adapted to the operational environment.
Mexico Country Strategy Evaluation
Evaluation of Mexico country operation found that UNHCR has been instrumental in expanding the protection space and contributing to the perception of Mexico as a host country for populations in need of international protection (rather than just one of transit and emigration). UNHCR and its partners have helped transform the national asylum system; there is broad consensus on the improvements achieved in access, registration, refugee processing and status determination, and local integration, despite systemic barriers and persistent challenges.
The evaluation recommends reinforcing coordination and regional analysis of population movements, capitalizing on existing data generation mechanisms and networks to drive evidence-based government and UN plans and responses. It also recommends promoting the strategic vision of the role to be played by the national asylum system among national institutions, within a context of intense mixed flows, in order to promote new migratory alternatives and preserve and broaden progress made in international protection and local integration.
UNHCR’s Approach to Accountability to Affected People
Synthesis of Evaluative Evidence
This report consolidates findings from independent evaluations for evidence of UNHCR’s multi-faceted approaches to Accountability to Affected People. This report aims to contribute to learning and reflection on a central tenet of UNHCR’s protection role and responsibilities: the participation by, and accountability to, the people UNHCR serves.
The evaluations find there is prolific use of a varied range of mechanisms and instruments by UNHCR operations to engage with and consult the beneficiary populations. Evaluations identify UNHCR’s wide presence in crisis-affected locations, and the operational need to engage directly with the people it serves, as important enabling factors for participatory activities. The evaluations revealed uneven levels of participatory practices in different operations and found limited evidence that the results of the participatory mechanisms resulted in changes in programming, implementation or prioritization. At the same time, evaluations note that UNHCR operations have made progress in establishing communication and information channels for affected populations, including those tailored to specific groups. UNHCR’s efforts and role are widely recognized by partners and counterparts.
6.The New Evaluation Policy
Our new Evaluation Policy builds on the Evaluation Policy of 2016. The vision going forwards is that:
“Evaluation informs choices made at all levels of the organization in strategic planning, programming and decision-making based on timely, credible and impartial evidence.”
In 2023, we will be evaluating UNHCR’s work in a number of important global strategic areas, such as engagement with internally displaced people, gender-based violence, decentralization and regionalization.
In addition, we plan to start the following country strategy evaluations in 2023:
Country Strategy Evaluations:
L3 Emergency response countries hosting refugees from Ukraine (Hungary, Moldova, Poland, Romania, and Slovakia) and some form of evaluative exercise inside Ukraine and for the Turkey and Syria earthquake response.
Examples of evaluations to be commissioned by Divisions/Regional Bureaus/Country Offices: