EOI - Consultant, BHA Ceton Rayuka II (Saving Lives) Final Evaluation at Catholic Relief Services

Catholic Relief Services is the official international humanitarian agency of the Catholic community in the United States. CRS works to save, protect, and transform lives in need in more than 100 countries, without regard to race, religion or nationality. CRS’ relief and development work are accomplished through programs of emergency response, HIV, health, agriculture, education, microfinance, and peacebuilding.

We are recruiting to fill the position below:

Title: Expression of Interest: Consultancy BHA Ceton Rayuka II (Saving Lives) Final Evaluation

Location: Maiduguri (Kaga, Gubio and Magumeri), Borno
Type of evaluation: End of Project Evaluation
Evaluation commissioned by: Catholic Relief Services (CRS), Borno State, Northeast Nigeria
Time frame of evaluation: June 2022 to August 2023 (Including 3 Months NCE)
Date: 25 July to 31 August 2023

Background

• Since the crisis in Northeast Nigeria escalated in 2013, the food security situation has continued to worsen. The Food Security Sector gap report in December 2022, Sector partners assisted 1.6 million people with food assistance (through in-kind or CVA) in the three most affected North-East states of Nigeria. The food assistance gap in the BAY states is 2.5 million people[1].
• Due to protracted security threats, a growing number of people are in urgent need of food assistance, and alarming rates of acute malnutrition have been consistently reported. In this emergency context, internationally displaced persons (IDP) households are among the worst affected populations as displacement has resulted in the loss and destruction of key productive assets and the interruption of livelihoods.
• Furthermore, in Maiduguri, the majority of IDPs families have found refuge with host families, with whom they share already exhausted resources such as shelter, land, and water.
• In responding to the protracted crisis, CRS Emergency and Early Recovery Program in northeast Nigeria focuses on food security and livelihoods, Nutrition, WASH, and shelter. CRS target the most vulnerable families for both IDPs and host families.
• To ensure coordination, effectiveness and efficiency CRS adopted a multi-sectoral approach for the provision of a comprehensive support that can assist vulnerable individual to realize minimum threshold of living standards.
• CRS attach high value to the adoption of community participatory practices in delivery of all intervention to enhance ownership. CRS adopted market-based approaches through e-vouchers using the Red Rose One System platform – leveraging integrated programming and empowering targeted communities through participatory approaches.

Scope of the Evaluation

• This evaluation will be implemented in three Local Government Areas (Gubio, Kaga and Magumeri) in Borno state. The evaluation will cover all the 3 sectoral foci of the project.
• The project targeted vulnerable community members, water facilities in the communities, and community leaders. Though the entire Household was the focus of the project, women were the primary target during registration.
• The consultant will conduct a preliminary desk review of project documents and monitoring records. With support from program leads, the evaluation will involve identification of the relevant stakeholders and beneficiaries, as well as collection of the required data to answer the evaluation questions.
• Some of the stakeholders engaged during the CRII include the community leaders, youth groups, and Bulamas in the three implementing LGAs, NGOs, BHA, etc.

The project target sites are:

Evaluation Timing and Duration

• The evaluation would commence in July 2023 with a desk research and field data collection and be finalized in August 2023, this will be preceded by validation of findings and preparation of draft reports.

Purpose of the Evaluation

• The overall objective of this evaluation is to assess the progress made and effects produced by the Ceton Rayuka II (Saving Lives) project across its entire life (from the inception to closeout). This may include assessing the change in food security status, reduction in hunger scores after receiving food assistance, different food security coping strategies adopted by communities, change in the nutritional status of malnourished children supported, knowledge, practices and skills retention for all training, sensitizations and counselling on optimal MIYCN-E practices conducted, improvement in utilization of water and sanitation facilities; and change in hygiene behavior and practices.
• For the purpose of quantifying the changes produced by the project, the evaluation will draw on evidence generated from the baseline survey and other related previous surveys and will be guided by these criteria: overall achievement, relevance, efficiency, effectiveness, coherence/ coordination, and indicative sustainability.

Evaluation Objectives
The evaluation will focus on achieving the following objectives:

• Assess the relevance, efficiency, effectiveness, coordination, and indicative sustainability of CRII interventions.
• Measure the overall achievement of the project against the targets in the results framework (project goal, sector objectives - outcome and outputs)
• Conduct Nutrition KAP survey to gather information on knowledge, attitudes, beliefs and practices among mothers/caregivers on MIYCN in Gubio, Kaga, and Magumeri LGAs.
• To determine and evaluate the complementarities of CRII sectors in achieving the overall goal.
• To identify lesson learned and actionable recommendations for future programming.

Indicative Evaluation Criteria and Questions

• The Consultant will work with the CRS MEAL and Project Technical Team to co-create an Evaluation Matrix (using this preferred template - MS Word) that will inform field data collection, management and reporting for the evaluation. The following is an indicative list of the evaluation questions to guide the design of the study.
• The successful consultant(s) can review, redefine, comment, or adjust the evaluation questions in their proposal where necessary to inform further discussions with CRS MEAL, Technical team, and BHA as part of inception consultation

Methodology of Final Evaluation

• Based on the scope and of the evaluation in accordance with CRS MPP policy and BHA guidelines, the consultant will be responsible objectives for developing a detail methodology, tools, and approaches for this evaluation. The evaluation will use a mixed method design, specifically an explanatory sequential design.
• This approach allows for the collection and analysis of both quantitative and qualitative data to gain a more comprehensive understanding of the observed changes.
• The design will follow a sequential process, with the quantitative data being collected and analyzed first to identify any patterns or trends.
• The subsequent qualitative data collection and analysis will then be used to explain the observed changes and provide context to the quantitative results.
• The quantitative component of the evaluation will, in addition to collecting primary HH-level, beneficiary-based cross-sectional endline survey data for Food Assistance and WASH indicators, involve reviewing the baseline data and other available data sources for the endline results of Nutrition indicators.
• The output of the quantitative arm of the study will be a comprehensive presentation of the baseline and endline values for all indicators against the approved targets in the project indicator tracking table across the three sectors. The difference between baseline and end-line values will be tested for statistical significance and discussed where it is possible.
• The qualitative component will use participatory qualitative approaches to gain sufficient in-depth understanding of the contextual factors and the results of the multi-sectoral assistance, as well as the perceptions of the key stakeholders. These two approaches will be complemented by content review and analysis of key project documents, deliverables, and other products.

Data Collection

• The Evaluator will conduct field data collection, CRS will provide the enumerators who will be trained by the consultant who could also make provision for supervisors to coordinating activities during data collection by the enumerators on the field.
• Due to access issues in some of the areas, remote data collection may be considered with strict compliance to CRS remote data collection modalities and BHA guidelines.
• The Evaluator will also use baseline and other surveys datasets and compare with the end-line values for each outcome indicator.
• For the output indicators, the Evaluator will run a trend analysis to assess the key points of change and use qualitative data to triangulate the changes observed.
• The Evaluator will use the BHA Ceton Rayuka M&E plan to guide decisions on specific indicator sampling frame.

Some indicative data collection approaches could include:

• Survey: Data to be collected using a survey questionnaire. The questionnaire must be designed relating but not limited to questions developed during the baseline and should be structured per sector in order to make comparison of the change observed from the baseline. For standardization of question interpretation, the evaluator should extensively train enumerators for them to conceptualize the questions in the local languages and training should incorporate practical demonstration. The consultants shall propose an acceptable sampling methodology (Size, approach and distribution) as part of the technical proposal.
• Desk review: review of strategic and planning documents, baseline report, semi-annual report, monthly report, PDM, pre and post phase assessment reports, sector dashboard, extract reports from red rose and Commcare platform and other related documents.
• Key Informants (KII) and focus group discussions (FGD): the evaluation team is expected to conduct key informant interviews and focus groups with key stakeholders in person or remotely. Key stakeholders will include, but not be limited to project beneficiaries, representatives of community leaders, local authorities, and representatives from project teams directly involved in the coordination of project implementation.
• Direct observation during data collection: Evaluators are expected to apply observation technique on children participated in Nutrition project, WASH facilities, hygiene practices and other factors related to the project. An observation checklist should be developed and agreed with the program managers before field data collection.
• The consultant is expected to review the indicative list of approaches described above and propose alternative or additional data collection approaches in line with BHA Monitoring and Evaluation guideline,

Data Protection

• CRS has ownership to all data collected during this evaluation.
• Data collected using paper-based tools are kept by CRS in a secure location to be accessed only by authorized persons. Electronic data is stored on cloud-based servers which vary depending on the software used for collection: Red Rose Collect data is hosted by Red Rose; Kobo Collect and PowerBi dashboard. Data collected on these platforms can be accessed by authorized MEAL Coordinator and PMs.
• The data will be used primarily for CRS final evaluation and future reference. Only findings from the analysis will be shared with the project partners, donors, and stakeholders. De-identified datasets will be uploaded on CRS Gateway (CRS’ internal knowledge management platform) as well as donor platforms such as USAID Development Experience Clearing House and Development Data Library.
• The consultant is obligated to conduct this evaluation in line with CRS’ Responsible Data Values and Principles and in full compliance with the Nigeria Data Protection Regulation(2019).

Ethical standards:

• The consultant is expected to follow American Evaluation Association’s Guiding Principles for Evaluators (Evaluation Guidelines and Principles - PDF)
• Prior to accessing CRS operational areas, the consultant, and his team, including Enumerators will sign CRS Code of Conduct, Safeguarding Policy, and Data Protection Policy.
• The consultant must seek and have a signed informed consent from participants before data collection with beneficiaries.
• The evaluator is expected to keep all data from beneficiary with anonymity and confidentiality, no mentioning of names in questionnaires or reports. and ensure protection of minors.
• The consultant will be expected to use locally acceptable conduct, dressing and norms to ensure the data collection is done in a respectful and dignified manner.
• Collected data can be stored on devices belonging to the CRS, but upon completion of the evaluation, all devices must be submitted to CRS.

Documents:
CRS will make available the following documents to the consultant:

• Project Proposals (inclusive of results framework, M&E Plan, organigram, etc.).
• Project Narrative report – monthly and annual
• Security - monthly reports.
• Previous Baseline and Evaluation reports.
• Post Distribution Monitoring Reports.
• Post-Harvest survey report
• Food Security Cluster reports.

Expected Deliverable
The evaluation team is expected to execute the following tasks:

• Inception report with a detailed Statement of work (indicative contents described earlier) for the evaluation, an evaluation matrix and methodology which respond to the agreed evaluation questions.
• Develop data collection tools and schedule for the evaluation. The survey data must be deployed through the Ceton Rayuka’s Commcare Space
• Train enumerators on data collection methodology and tools
• Facilitate completion of data collection and ensure datasets are uploaded for all the interviews conducted in line with CRS data policies.
• Clean and conduct a rolling analysis (using Power BI) of the survey data as the data collection progresses and submit the Power BI file along with the preliminary report for verification by CRS MEAL team.
• Develop and/or modify qualitative data collection tools in line with the findings from the quantitative survey.
• Collect and analyze the qualitative data using any preferred software and submit the analysis code book as part of the final report.
• A comprehensive evaluation report which responds to the evaluation questions for each sector including written responses to comments/feedback from CRS staff.
• Produce a PowerPoint presentation with key findings to be presented during the evaluation briefing/reflection event.
• Develop a 2-pager evaluation infographic to include relevant findings, key points, and recommendations.
• Maternal Infant and Young Child Nutrition (MIYCN) knowledge, Attitude, and Practices (KAP) survey report (Findings).

Requirements for the Final Evaluation Report are as follows:

• Maximum 30 pages excluding annexes in Microsoft Word and PDF format for the approved version.
• Written in English.
• Sections:
  • Title page
  • List of acronyms and abbreviations
  • Table of contents, including a list of annexes.
  • Executive summary - (max. 1 and half page) including bullets of main findings that answer the evaluation question and fulfill the evaluation purpose.
  • Introduction describing the program’s background and context.
  • Evaluation purpose and questions
  • Methodology (max.1pages)
  • Sector specific key findings and detailed analysis organized by evaluation question and responding to theory of change.
  • Recommendations – Sector-based recommendations based on the evaluation finding.
• Annexes:
  • Relevant maps
  • Annotated photographs with consent for publication where relevant
  • List of existing data
  • Finalized data collection tools.
  • List of interviewees from primary data collection
  • Raw datasets, transcribed notes from FGDs and KII and observation notes
  • File containing analyses of data, both end-line and primary data collection.

Intended User:

• The primary users of this evaluation will be CRS Project Team including regional-level Program Development and Quality Assurance team, Emergency Coordinator, all sector Program Managers, Monitoring, Evaluation, Accountability, and Learning Coordinator and Team, and Nigeria Country Program Management.
• Findings of the evaluation will be discussed with beneficiaries, local partners and stakeholders who were directly involved in the implementation of the project in a reflection event (in line with CRS MEAL Policies and Procedure 3.7).
• Finally, the report will be shared with the donor (USAID) to communicate the project’s contribution to the change achieved in emergency response in Borno state.
• Findings from this evaluation may inform the technical design and implementation of future, similar projects. In addition, the report will also document lessons learned and good practices for implementing projects in emergency context that can be replicated in other CRS programs; and may also help identify opportunities to enhance implementation strategies in future.

Evaluation Team Composition

• The evaluation is expected to be conducted by two evaluators: a lead evaluator and assistant evaluator who will oversee the execution of the entire evaluation process in partnership with CRS staff.
• The MEAL Coordinator will lead the overall coordination of the evaluation, checking quality and compliance issues to the CRS and USAID evaluation guidelines.
• The evaluator will work with support from enumerators from CRS for data collection. Clear roles and responsibilities are outlined in the table below.

Timeline and Milestones
Date - Milestone:

• July 25th 2023 - Inception report, methodology, workplan and sampling frame
• August 2nd to 4th August, 2023 - Enumerators’ training
• August 7th to 11th August 2023 - Data collection
• August 12th to 22ndAugust 2023 - Data analysis and report writing
• August 23rd to August 25th 2023 - Submission of final evaluation report to CRS
• August 26th to August 29th 2023 - CRS review report and provide feedback
• August 31st 2023 - Final evaluation report and brief submitted and approved

Required Qualifications
Lead Evaluator:

• Post graduate qualification in Development Studies, Monitoring and Evaluation, Statistics, or International Development, PhD will be an added advantage.
• Extensive knowledge of evaluating multisectoral projects in emergencies and good understanding of the emergence response in the Northeast.
• At least 5 years’ experience in conducting evaluations in complex environments, experience evaluating USAID/BHA funded projects. Experience working in Nigeria Northeast states, preferably BAY states will be an added asset.
• Extensive experience in evaluating multi-sectoral programs implemented through multiple partners.
• Excellent oral and written communication skills in English and fluency in Hausa and Kanuri.
• Extensive knowledge of qualitative and quantitative data collection methods and analytical methods and techniques.
• Experience in generating strategic, informative, and action-oriented recommendations to CRS project staff and strategic management.

Application Closing Date
25th May, 2023 by 5:00pm Close of Business (West Africa Time).

How to Apply
Interested and qualified candidates should send full Applications electronically to: [email protected] with the subject line “RFQNG11129 Application for BHA Ceton Rayuka II – Saving Lives Final Evaluation”.

Application Process

• Applicants when applying should submit a brief (maximum 3 pages) approach paper outlining how they understand the TOR and proposed methodology.
• In addition, applicants should submit their profile highlighting key areas of expertise and experience matching the required skills specified in these terms of reference.
• We also request links to at least three recent examples of relevant work, and the contact details for three references.
• Indicative budgets should be prepared in line with the expected deliverables and should show all related costs including travel costs and anticipated overhead costs.