MHPSS Evaluation in Protracted Crisis: Field Notes from Palestine
Applying OECD-DAC criteria in active conflict demands more than methodological rigour — it requires ethical adaptation, clinical team composition, and a willingness to let programme context reshape evaluation design.
Evaluating MHPSS programmes in active conflict is methodologically demanding in ways that standard OECD-DAC guidance does not fully anticipate. This note draws on two programme evaluations conducted across Gaza, East Jerusalem, and the West Bank to offer practical field notes for evaluation practitioners working in similar contexts.
The OECD-DAC criteria don't break — but they need adaptation
Relevance, effectiveness, efficiency, impact, and sustainability remain the right framework. What changes in a protracted crisis is how each criterion is operationalised. Relevance requires asking whether the programme is responding to the right presenting needs at this moment — which shifts rapidly in active conflict. A programme designed for chronic complex trauma looks very different from one responding to mass casualty events. We found it useful to build explicit sensitivity to programme adaptation into our evaluation questions: not just did it work, but how did the programme change in response to the crisis, and were those changes appropriate? Sustainability, similarly, asks different questions when institutional continuity is genuinely uncertain. We found it more useful to evaluate sustainability at the level of trained workforce and community-embedded approaches than at the level of organisational infrastructure.
Staff wellbeing as a substantive evaluative indicator
In both evaluations we treated staff wellbeing as a substantive indicator — not an afterthought. Practitioners delivering MHPSS services in Gaza post-October 2023 were simultaneously affected community members. Their capacity to deliver quality services, maintain professional boundaries, and manage caseloads was directly affected by their own exposure to trauma. Staff interviews were conducted by team members with clinical MHPSS backgrounds, in private, with clear separation from any performance review process. The findings — on supervision adequacy, peer support structures, and caseload manageability — became some of the evaluation's most actionable outputs, with concrete recommendations for management that programme teams could act on immediately.
Ethical data collection with conflict-affected populations
Standard informed consent procedures require adjustment when respondents may associate formal paperwork with risk. We simplified consent forms and used verbal consent in all FGD settings, building in additional rapport time before structured data collection began. Beneficiary interviews were conducted in trusted community settings rather than programme facilities wherever possible. All data on sensitive indicators — trauma symptoms, GBV disclosure history, suicidal ideation in adolescent programming — was collected exclusively by trained clinical staff, never by data enumerators, and transmitted under strict data protection protocols.
Referral pathway assessment as a core effectiveness question
Referral pathway functionality emerged as a central effectiveness question in both evaluations. The most technically sound MHPSS programme fails beneficiaries if the systems around it cannot provide the specialised care that case management identifies. We assessed pathways through case file review, KIIs with receiving facilities, and — where accessible — traced case interviews. A consistent finding: referral documentation was stronger than referral outcomes. Recording a referral and confirming a beneficiary received the referred service are not the same thing, and MEAL systems rarely track the latter. This gap represents one of the most significant quality risks in MHPSS case management, and one of the most addressable.
On team composition
MHPSS evaluations are strengthened by genuine multidisciplinarity. Our teams combined evaluation methodology expertise with clinical MHPSS knowledge and deep contextual familiarity with Palestine. That combination shaped not just the quality of data collection but the interpretation of findings — and ultimately the utility of recommendations for programme teams navigating genuinely difficult operating environments.
Building MEAL Systems That Work in the Field
The gap between a well-designed MEAL system and one field teams actually use is one of the most persistent failures in development programme management. This note draws on a regional MIS implementation connecting operations across three countries.
MEAL systems most often fail not because of technical inadequacy but because they were designed around donor reporting requirements rather than programme team decision-making needs. This note documents lessons from a nine-phase regional MIS implementation connecting UPA's operations across Palestine, Jordan, and Lebanon — and from the patterns we encounter when inheriting systems built without that distinction in mind.
The most common failure mode
When the primary output of a MEAL system is a quarterly donor report, field teams quickly learn the system exists for compliance — not for them. Data quality declines, workarounds proliferate, and the system gradually becomes a parallel administrative burden rather than a management tool. The fix is to design data flows around the questions programme managers actually want answered, and treat donor reporting as a downstream output of that — not the primary driver. In practice, this means sitting with programme teams before system design begins and asking: what do you genuinely need to know each week to manage this programme well? Those answers drive the system architecture.
ActivityInfo as a field data backbone
For organisations operating across MENA, ActivityInfo has become our preferred platform for field-level data collection and indicator tracking. Its offline functionality handles the connectivity constraints common in field settings; its form builder is accessible to non-technical staff; and its outputs feed directly into Power BI for dashboard visualisation. In the UPA implementation, we connected data from three country offices — each with different programme structures and indicator frameworks — into a unified data architecture. The upfront investment in indicator harmonisation and data governance protocols paid off in cross-country real-time programme data, replacing a quarterly Excel reconciliation process that was absorbing significant M&E officer time.
Power BI dashboards — the choices that matter
A Power BI dashboard is only useful if programme teams actually open it. That sounds obvious, but a significant proportion of dashboards we inherit have not been accessed by anyone other than the M&E officer in months. Useful dashboards are designed around three or four genuinely important management questions. They are not comprehensive, do not attempt to visualise every logframe indicator, and are structured around how programme teams think about their work — not how a results framework is sequenced. Load time, screen size, and language all matter more than visual sophistication. We also invest heavily in the handover process: a dashboard a team understands and trusts is far more valuable than a technically sophisticated one that functions as a black box.
The governance question nobody asks early enough
In multi-country or multi-partner implementations, data governance is the issue that causes the most downstream problems when not addressed at the design stage. Who has permission to enter and edit records? Who can view aggregated cross-site data, and who sees only their own? What happens to data access when a staff member leaves? What is the protocol when submitted data is contested? We document data governance in a formal protocol and build permission structures into the system architecture from the start — not as an afterthought when a data breach or a disputed entry first surfaces.
Evaluation Under Constraint: Participatory Methods in Restricted Contexts
Two engagements — a food security evaluation in Area C and a six-ministry ECD strategy process — present different methodological challenges with a common thread: useful evaluation evidence requires adapting methods to context, not forcing context to fit methods.
Two of our recent engagements — a final evaluation of a food sovereignty programme in Beit Jala (Area C, West Bank) and the development of Palestine's National ECD Strategy spanning six government ministries — presented very different methodological challenges. What connects them is a principle that holds across both: useful evaluation evidence requires adapting methods to context, not forcing context to fit a preferred method.
Area C: treating access as a live methodological variable
Evaluation in Area C of the West Bank requires treating access as a live methodological variable rather than a fixed condition. Entry restrictions, unpredictable road closures, and community sensitivity to external presence all affect what data can be collected, from whom, and when. For the Beit Jala evaluation, field observations were conducted in pairs with community guides rather than by external evaluators alone. FGD timing and locations were determined by community leaders, not by our data collection schedule. The evaluation team included local researchers with existing community relationships — not as a courtesy, but as a methodological necessity. These adaptations produced richer data than a more externally controlled process would have. Community members spoke more openly in settings they controlled, about a programme they had participated in designing.
ATLAS.ti for rigorous qualitative analysis
With 1,495 beneficiaries and a programme embedded in a specific community context, the qualitative dimension of the evaluation required systematic analysis across a substantial volume of transcript data. We coded all FGD and KII transcripts in ATLAS.ti using a coding framework developed iteratively from both the evaluation questions and the data itself. This approach allows us to move from individual responses to patterns across the full dataset in a way that is rigorous and auditable — commissioners can follow the analytical trail from raw transcript to published finding. It also surfaces divergences between beneficiary accounts and programme documentation that would be easy to miss in a less systematic process. In this evaluation, those divergences became some of the most important findings.
Multi-ministry participatory processes
The National ECD Strategy process presented an entirely different challenge: not restricted access to a community, but the coordination of six government ministries with overlapping mandates, political relationships, and institutional cultures. Participatory strategy development at this scale requires deliberate facilitation design. We used bilateral ministry consultations before any joint sessions to understand each ministry's priorities without the political dynamics of a shared table; structured joint workshops with deliberate role design to prevent any single ministry from dominating; and iterative draft review processes that allowed disagreements to surface and be negotiated before they became blocking issues. The resulting strategy reflected genuine multi-ministry ownership — because the process was designed to create the conditions for it, not assume them.
A principle that holds across both contexts
In both engagements, what produced the most useful evaluation evidence was genuine participant ownership of the process. In Beit Jala, community members shaped when and how data was collected. Across the ministries, government counterparts shaped what questions the strategy needed to answer. This is not simply an ethical position — though it is that. Evaluation evidence is more useful when the people who generated it feel ownership of the findings. Utilisation is downstream of participation, and participation requires more than consultation.