Measuring, Learning and Adapting

Introduction

Recognizing the value of a responsive and adaptable approach is central to the Sustainable Development Goals and to UNICEF’s 2022-2025 Strategic Plan. A flexible and agile approach allows for course correction in the face of volatile environments, new evidence and insights from implementation. 

This adaptive management approach aligns well with our work in Social and Behaviour Change by allowing us to design interventions that are tailored to the context and accommodate complexity. Embracing iteration and constant learning should be at the heart of our work.

Iteration assumes a learning mindset. Making improvements and adjustments to strategy design based on new evidence and contributions is an expected and central part of the process. We assess, learn and adapt through a regular cycle of rapid feedback and adjustments. Each cycle gets us closer to solutions that effectively address the challenge we seek to solve. Tackling complex social and behavioural problems requires significant input from the people and communities we are partnering with before implementation is scaled up.

Embracing the notion that we do not have all the answers is fundamental to an iterative approach. Local needs and strengths need to be discovered by and with communities and partners. Strategies should change over the course of a programme’s lifespan to accommodate evolving needs and context. It is also important to consider that people may not realize what they need until it is made tangible. This uncertainty is at the heart of an iterative process.

Rapid revision and feedback loops can increase ownership, efficiency and cost-effectiveness, decrease risk and manage uncertainty, all while encouraging continual improvement.

How is iteration different?

Traditional project approaches are built on a linear, cause-effect framework, with every step contributing to an end result. Many development initiatives and donor expectations have relied on this horizontal design, with progress divided into stages along a project lifecycle: a research phase, a project design phase and a delivery phase – with monitoring moments throughout.

At the end of this sequence of phases, an evaluation is conducted to determine whether the initiative reached its objectives.

Unlike the sequential approach, the iterative process plans for many moments of failure and reflection.  It acknowledges that our work, and the world in which we do it, is imperfect. What people say and do are often different things. Unanticipated challenges are inevitable throughout the course of any programme. Decisions made in the design phase are likely to change before the final report is delivered.

The traditional approach is less dynamic and responsive to unexpected changes in context or needs. It is not designed for quick modifications to address inefficiencies on the fly. This rigidity means that changes end up being more costly and results remain untested and uncertain until the very end.

Allowing for iteration in our work offers flexibility. It can help to identify weaknesses in project design and in the premise of the work itself. It can point out where projects are inefficient, ineffective and missing the mark. This helps counter sunk cost fallacy – the idea that we need to follow through on something we have started, even if evidence shows there will be limited benefit.

A note on targets and indicators

Though the primary focus of the strategy, expressed through the problem and opportunity statements, should be maintained, an iterative process may reveal unexpected ways to address it. However, it may not be as simple as tracking progress towards intermediate results and indicators established at the outset. You may discover that you were measuring the wrong things. We do not know all the answers at the beginning, so how we measure success is likely to change over the course of the project.

For example, a project focussed on increasing secondary school enrolment for girls may begin by addressing awareness of the benefits of education. Testing this assumption throughout the project may reveal that the problem has nothing to do with awareness. Parents may understand the importance of school, and girls may want to attend. However, you may discover that societal pressures to marry, pregnancy issues or the need to financially support one’s family are far more significant barriers to enrolment. While increased enrolment remains the objective, the focus of the interventions and what we measure will change with new insights gathered through ongoing monitoring, learning and adapting.

Steps for an iterative process

Measuring, learning and adapting should be continuous throughout the lifecycle of the strategy.

Opportunities for iteration

Key milestones are good opportunities to stop and take stock of new knowledge and learnings and to reflect upon and change course, where appropriate.

The overall strategy or programme can be designed with more flexibility and responsiveness. However, activities can also embrace agile and rapid cycles of learning and adaptation at the micro level. This can be broken down into four steps: 

1. Plan and confirm objectives  
   Always refer back to the issue the iteration is meant to address. You may need to break the project down into smaller component parts that provide multiple moments for input, testing and feedback. These research ‘sprints’ can provide valuable insight into key moments along the project lifecycle.  
    
2. Create your first iteration  
   Iterations do not need to be fully realized prototypes. They can be rough physical mock-ups, paper sketches and maps, lo-fi examples of digital tools, service adjustments, changes to the physical space, etc. They can also be strategies that infuse flexibility, rapid feedback and adaptation from the beginning. The goal is to rapidly test your hypotheses and gather evidence.  
    
3. Test  
   Gather feedback from stakeholders and partners through structured surveys, focus groups or other tactics. Use this opportunity to realign with your original objectives – what you hope to achieve and how your early ideas are being received.  
    
4. Evaluate and review  
   You should now have information you can act upon.  It is important to keep an open mind. This is not a moment to decide whether something has succeeded or failed. Instead, consider the following questions: Does this iteration meet your objectives? What do we know now that will make this better? Continue adapting your approach and design until you get to a next-generation iteration.  
   Remember, iterative development can take weeks or even months. Let your project objectives be your compass - revisit them every time you begin a new cycle.  

When should this process end?  

The amount of iteration depends on the initiative. It is important that your work is flexible enough to consider evolving circumstances, changes in context and direct feedback from the communities you are working with. This will help you determine whether new learnings will have  a meaningful and measurable impact on your work. You should expect fewer cycles over time.

A successful iterative process should:  

• Increase quality and functionality. Each iteration should get you closer to achieving your goal. If feedback suggests you are off-track after several rounds of iteration, analyze the feedback to determine how you might course correct.   
• Reduce levels of change. With each iteration, the amount of fundamental changes should decrease. Iterations should decrease in scope over time from large shifts in approach to minor tweaks..  
• Increase fidelity. Ideas will likely become more sophisticated as they solidify. An app originally designed on an A4 sheet of paper may evolve to a cardboard prototype to a working digital draft to a phone-based prototype. A social change strategy will become more defined with every community working session and small-scale roll-out in pilot communities.   
• Increase sense of the risks. The iterative process may reveal more areas of risk. Expect to learn more about where your efforts may fail than where they may succeed. Iteration may reveal layers of complexity that were undetected during the design phase.  
• Measure and document progress. Capture and record any adjustments you make toward an improved outcome for future reference. 

Is iteration always the answer?

Iteration can increase efficiency, promote collaboration, help identify risks and, ideally, lead to suitable, effective and appropriate strategies. However, iteration also comes with several risks:

1. Expectations of donors, partners and colleagues. Traditional project management relies on predictable phases, budgets and outcomes, which assumes fixed timelines, finances and capacity. In order to uncover new opportunities and challenge assumptions, an iterative process invites  delays and changes in focus or budget. These can be seen as failures or shortcomings in traditional project design and monitoring. Donors, colleagues and partners may have more rigid expectations. Iteration should ideally be planned for and adequately resourced. Without managing these expectations up front, iteration can be seen as inefficient, ineffective and expensive. 
2. Unclear timelines. There is no predicting when you will arrive at something that works. It is difficult to define the degree of changes, the length of testing and the time it takes to address feedback. Not knowing when to stop iteration is also a risk.   
3. Scope creep. Constant learning and adaptation is essential to iteration. However, this can reveal areas, features, problems and opportunities that you didn’t expect when the programme was being designed. A capacity-building project may ultimately result in a fundamental change to supervisory arrangements at community health centres. An awareness campaign may lead to the realization that all the awareness in the world will not incite change without better resources from the central government. 

UNICEF’s Adaptive Programming Core Team highlights key programming principles – preconditions to guide this shift across programmes. They include: 

• An enabling organizational culture of experimentation, innovation, documentation and learning that incentives staff to build on success, learn from failure and share all insights equally, without fear of recrimination.  
• Openness to a more iterative approach to planning, implementation, monitoring and ongoing reflection and the willingness and ability to adjust plans and interventions based on new insights and knowledge, while remaining focused on outcomes.  
• Commitment to building a dynamic, living evidence base comprising formal and informal evidence, quantitative and qualitative data, stories, case studies, examples and learnings, drawn from both programme-generated evidence and the wider global evidence base.  
• A larger emphasis on real-time programme monitoring enabling more timely course correction, learning and knowledge sharing. All tools, guidance and systems accompanying such shifts should also remain agile.  
   

For a full list of recommended principles, check out this discussion paper

Case studies and examples

Monitoring during the project lifecycle

• KENYA: A partnership between WSUP, GAIN, Aqua for All, Unilever and Ideo came together to find a solution to the poor accessibility to clean drinking water that affects 39% of Kenyans. The project used an agile process of continuous action research, prototyping, feedback and iteration to find a solution that worked for intended groups at the ground level, and at a system level.
• KENYA AND ZAMBIA: PATH’s Living Labs led an iterative process, using HCD and co-creative methodologies to design a COVID-19 vaccine distribution plan with the frontline health care workers ultimately delivering the service. The team used prototyping in the first round of research and an online survey to collect feedback on a label design. They then used focus groups to test an iteration with higher fidelity packaging and vials. Findings from this effort were then prioritized by healthcare workers. They developed a model universal packaging and labelling system to to support effective service delivery when packaging is not immediately available or informational leaflets are missing. 

Evaluating impact post-implementation

• GLOBAL: UNHCR’s Digital Inclusion programme tested the feasibility of engaging communities using WhatsApp, a key messaging platform for many communities across cultures, to address privacy risks of digital communication.  In collaboration with Praekelt and its Turn.io product, which facilitates large-scale education programmes via WhatsApp, they designed a piloting phase to assess the digital risks and data protection issues of this medium. UNHCR has made available its report, complete with recommendations on how humanitarian organizations can use messaging apps to engage with crisis-affected people.

Incorporating changing behavioural needs into your solution

• TANZANIA: The Twaweza programme provides teenage girls with a curriculum to improve menstrual health and hygiene by providing Femme Kits containing their choice of sanitary products – either washable pads or a menstrual cup. The programme aims to improve school attendance by helping those who do not have access to hygienic, sustainable menstruation management methods to leave the home with confidence. The Impact Report describes a 70% uptake of menstruation cups and a 20% decrease in the use of fabric pads, demonstrating the curriclum’s impact on the girls’ choices. The implementation of this service is designed with a mechanism to respond to changes in the behaviours that the programme set out to achieve. Creating space for iteration at the end of the project timeline has enabled the programme to provide the right products at the right time, leading to its overwhelming success.

Key resources  

    ➔   UNICEF Discussion Paper:   Developing a Systematic Adaptive Programming Approach to Support UNICEF’s Strategic Plan 2022-2025  
    ➔   Adaptive Management: Learning and Action Approaches to Implementing Norms-shifting Interventions 
    ➔   A Prototyping and Feedback tool and Piloting and Iteration tool from UNICEF’s HCD for Healthcare   
    ➔   A 90-minute Prototyping Activity from Hyper Island, part of its open-access Innovation Toolkit