PO6311 Independent Evaluation for Promoting Pro-poor Opportunities in Commodities and Service Markets (Propcom) Mai- karfi Programme, Northern Nigeria Draft Interim Phase 2018 Evaluation Report – Poultry Vaccination Intervention April 2018 PO6311 Independent Evaluation of Propcom Mai-karfi – Draft Interim Phase 2018 Evaluation Report – April 2018 Table of Contents 1. Summary of Findings ....................................................................................................................... 1 2. Introduction and Context .................................................................................................................. 2 3. Intervention Conceptualisation and Design ..................................................................................... 2 4. Rationale for the Evaluation............................................................................................................. 3 5. Evaluation Methodology .................................................................................................................. 4 6. Limitations to this Evaluation ........................................................................................................... 5 7. Findings ........................................................................................................................................... 7 7.1 Relevance ...................................................................................................................................... 7 7.2 Effectiveness ................................................................................................................................. 9 7.3 Efficiency ..................................................................................................................................... 11 7.4 Sustainability ................................................................................................................................ 12 7.5 Impact .......................................................................................................................................... 15 8. Conclusions ................................................................................................................................... 17 PO6311 Independent Evaluation of Propcom Mai-karfi – Draft Interim Phase 2018 Evaluation Report – April 2018 List of Acronyms AAER Adopt Adapt Expand Respond ACI Agric-project Concept International CAHW Community Animal Health Worker CBO Community Based Organisation FGD Focus Group Discussion KII Key Informant Interviews MADE Market Development in the North of Ghana Independent Evaluation NCD Newcastle Disease NVRI National Veterinary Research Institute OECD DAC Organisation for Economic Co-operation and Development's Development Assistance Committee Propcom Promoting Pro-Poor Opportunities through Commodity and Service Markets Mai-karfi Programme PM Propcom Project Manager VBI Village Based Inoculator VBV Village Based Vaccinator VCN Veterinary Council Nigeria VP Village Promoter WEE Women Economic Empowerment PO6311 Independent Evaluation of Propcom Mai-karfi – Draft Interim Phase 2018 Evaluation Report – April 2018 1 1. Summary of Findings The key objective of the Poultry Health intervention is to establish an efficient channel for the distribution of Newcastle Disease (NCD) vaccines to rural households, in order to decrease poultry mortality and contribute to increased household resilience and decreased poverty. This intervention is implemented by the Promoting Pro-Poor Opportunities through Commodity and Service Markets Mai-karfi Programme (Propcom) in partnership with Agric-project Concept International (ACI), a privately owned veterinary supply company based in Kaduna City in Kaduna State. After a pilot which mainly targeted women as vaccinators, the business model focussed on identifying individuals who understood and appreciated the cost recovery model and could see the income opportunities of providing vaccination services. While developing and refining this business model to overcome last mile distribution challenges in cooperation with ACI, Propcom Mai-karfi needed to address supply side constraints in the production of the vaccine and concerns of the regulatory bodies regarding the administration of vaccines by unlicensed and unregulated individuals. As well as demand side constraints of the provision of accurate, applicable and trustworthy information about the disease and its prevention. To overcome the former, the programme worked with the parastatal manufacturer of the vaccines, state agencies involved in educating about the disease and regulatory veterinary bodies regarding the standardisation and formalisation of the vaccinator training. To overcome demand side constraints, the programme leveraged on the state agencies communications campaigns, and developed standalone promotional and educational material that the various stakeholders in the distributions channels could easily use. An endline quantitative evaluation1 of the Poultry Health initiative speaks to the intervention impact at a household level and the extent to which this decreases household poverty and contributes to increased household resilience. However, there has been considerable learning in the design, development and implementation of this intervention and there has been impact on stakeholders throughout the market system. Various market system players are also displaying early indicators of systemic change. Headline learning includes: • The intervention was appropriate and relevant to the needs of the programme beneficiaries – poor households in northern Nigeria. W omen members of these households have benefited either indirectly as members of the household or directly as owners of household poultry. • The programme implemented the intervention in cooperation with numerous market system players, utilising and leveraging off existing initiatives. In turn, the programme shared its intervention learnings with other developmental programmes, allowing them to build and adapt the intervention for their own purposes. • Regulatory authorities are aware of the impact of the intervention and are supporting a more ambitious, wider roll out to incorporate more comprehensive veterinary services to the rural poor. • The model designed and adapted as part of the intervention has been further refined, adapted and adopted by other market system players. As a result, this intervention and others of a similar nature are likely to continue to be implemented in future. 1 Independent Evaluation for Promoting Pro-poor Opportunities in Commodities and Service Markets (Propcom) Mai-karfi Programme, Northern Nigeria, Endline Evaluation Draft Report – Poultry Health April 2018. PO6311 Independent Evaluation of Propcom Mai-karfi – Draft Interim Phase 2018 Evaluation Report – April 2018 2 2. Introduction and Context The keeping of chickens as an easy source of protein, as well as a form of household savings and income, is a widespread practice in northern Nigeria. Nearly two thirds (62%) of Nigeria’s poultry is raised in household backyard farming. On average, each farmer has less than 20 birds. Ownership of the birds might be regarded as household assets or it might be ascribed to an individual in the household. No matter the owner of the birds, the care of the flock generally falls to the women in the household. The owner of the birds generally claims the income from selling the poultry, regardless of who cares for the flock. Income is used for a variety of purposes, including school fees, medical expenses, and buying farming inputs such as fertilizer, but is often used for family emergencies, reflecting the ease of converting poultry to cash when needed. During the harmattan, farmers lose between 50‐100% of their poultry flock due to NCD, an annual phenomenon which can only be prevented through vaccination before the bird gets infected. The vast majority of the farmers have never vaccinated their birds due to limited access to information and no access to vaccines and veterinary services. Instead they opt to administer ineffective local treatments. While there are more than 30 companies of varying scale selling veterinary products in Nigeria, they tend to focus on commercial farmers, not appreciating rural free-range poultry owners as potential customers. The distributors and retailers (many of whom are veterinarian doctors) prefer to be located in the towns instead of rural areas. This is due to the combination of the availability of amenities, a larger customer base, the lack of a cost-effective distribution channel to reach rural communities, the costs of travel to rural areas for companies, as well as the general perception that small farmers are neither willing nor able to pay for services. 3. Intervention Conceptualisation and Design Propcom Mai-karfi’s intervention was designed to overcome a number of limitations within this market system including: lack of information regarding the disease and suitable treatment, lack of access to the vaccine (and to a vaccine that is heat tolerant), lack of service provision to administer the vaccines affordably and lack of an information and distribution network to the more outlying, rural areas. The intervention wove together numerous threads including training village based agents and engaging with state wide and federal regulatory bodies. This resulted in a high score for innovation, with Propcom Mai-karfi working with both regulatory issues as well as logistical and capacity restraints within the market system (See Figure 1: Vaccine Value Chain Showing Market System Challenge Areas). Figure 1: Vaccine Value Chain Showing Market System Challenge Areas PO6311 Independent Evaluation of Propcom Mai-karfi – Draft Interim Phase 2018 Evaluation Report – April 2018 3 The designed intervention aimed to train Village Based Vaccinators (VBVs), who earn an income from vaccinating chickens in their local communities. It was anticipated that vaccinations would not be a sole source of income for these individuals, and that they might also act as service providers for other veterinary inputs and services. ACI, a Propcom Mai-karfi private sector partner, would source and distribute the vaccines, often to established regional veterinary service providers and through facilitated relationships with Community Based Organisations (CBOs). These providers would act as distribution points for the VBVs. The National Veterinary Research Institute (NVRI) in Vom, Plateau State, would replicate and provide the vaccines to ACI. Propcom Mai-karfi faced early opposition to the intervention, and engaged with state and federal agencies to address these concerns. W orking with these agencies, the programme initially cooperated in the development and launching of NCD information campaigns and then subsequently in the development of a curriculum for training Community Animal Health Workers (CAHW). Through this cooperation the programme worked alongside the regulatory authorities to overcome their initial concerns and demonstrated the success and impact of the pilot roll outs. The intervention rationale was that the impact of improved access to vaccination services would increase demand for the vaccine which in turn would result in decreased poultry mortality within households, leading to an increase in household or individual assets and decreasing household poverty. 4. Rationale for the Evaluation At the programme level the independent evaluation comprises two quantitative evaluations and a number of qualitative evaluations. Poultry Health was identified relatively early in the evaluation process as one of the quantitative evaluations.2 Aspects of the poultry vaccination market intervention have also been the focus of one of the qualitative evaluations.3 Initial pilots of the intervention led to changes in the model, which have since proved successful within their areas of operation. Further to this, Propcom has made considerable strides in working with veterinarian regulatory authorities in this area, increasing the sustainability likelihood of the intervention. This multiple strand focus of the intervention sets it apart from a number of others, in that Propcom Mai- karfi is working with stakeholders at almost all levels of the market system. However, for the intervention to be firmly entrenched with the commercial partners it has to make commercial sense, and part of this rationale is achieving scale and efficiencies in distribution. These issues relating to scale and efficiency need to extend not only to the identified commercial partner but also to the vaccine distributors and to the VBVs. This qualitative study explores these and other issues related to scale and increased reach as well as the current challenges that the intervention faces in firmly embedding itself within the Nigerian commercial landscape. The report focuses on the area of the results chain identified below in Figure 2: Simplified Poultry Vaccination Results Chain. 2 Independent Evaluation for Promoting Pro-poor Opportunities in Commodities and Service Markets (Propcom) Mai-karfi Programme, Northern Nigeria Baseline Report - Quantitative Poultry Health Market Survey, December 2015. 3 Independent Evaluation for Promoting Pro-poor Opportunities in Commodities and Service Markets (Propcom) Mai-karfi Programme, Northern Nigeria, Interim Phase 2016 Evaluation Report, November 2016. PO6311 Independent Evaluation of Propcom Mai-karfi – Draft Interim Phase 2018 Evaluation Report – April 2018 4 Figure 2: Simplified Poultry Vaccination Results Chain 5. Evaluation Methodology After identifying the poultry vaccination market as a subject for further evaluation, the evaluation team developed a scoping note, outlying the purpose and range of the evaluation. The scoping note identified the number of interviews and focus groups that were proposed and the potential respondents in each of these. The scoping note was shared with the programme, with their comments and feedback incorporated into the evaluation design. During its inception phase the evaluation identified key evaluation questions. Each of the subsequent studies reflects on these questions and ties the study to specific questions. Some evaluation questions might not be reflected directly (e.g. the involvement of women) but are addressed in sub-questions during data gathering. The current evaluation focusses on six questions from the Evaluation Inception Report. These are linked to the Office of Economic Cooperation and Development (OECD) Development Assistance Committee (DAC) criteria, which provides the structure of the findings. The evaluation questions addressed in this research are: • Were the outcomes and impacts delivered relevant to beneficiaries needs? (Relevance in the DAC criteria.) • How effective was the programme in meeting its results promised for women and girls, who were only 7% of beneficiaries in Propcom 1? (Impact, effectiveness, efficiency and coverage and inclusivity in the DAC criteria.) • How effective was the programme in decreasing vulnerability to shocks, trends and seasonality of poor women and men engaged in selected rural markets? (Impact and sustainability in the DAC criteria) • How did the interventions interact with other factors in the local and regional economy? (Impact and sustainability in the DAC criteria.) • Were the interventions really catalytic and did this lead to systemic change resulting in better access to services and inputs? (Impact and sustainability in the DAC criteria.) PO6311 Independent Evaluation of Propcom Mai-karfi – Draft Interim Phase 2018 Evaluation Report – April 2018 5 • To what extent did the programme’s interventions stimulate the continuation in the market of existing players and the entry of other players into the market in a long term, positive and sustainable way? (Sustainability in the DAC criteria.) In the Inception Report, the noted OECD DAC criteria were allocated to each of these questions. The findings of this evaluation are reported according to the DAC criteria. Given that the programme is an M4P programme, the Findings Section 7 incorporates findings that relate to the Adopt Adapt Expand Respond (AAER) matrix.4 Focussing on these Inception Report questions, instruments for Key Informant Interviews (KIIs) and Focus Group Discussions (FGDs) were developed and shared with the programme for comment. Feedback from the programme was taken into consideration in the finalisation process, before piloting. Pilot instruments were shared with the local partner and refined to more accurately reflect wording and phrasing appropriate to the respondent audience. The local partner trained local enumerators in these instruments and subsequently piloted the instruments. Final adjustments were made to the instruments following the piloting process, and enumerators re-familiarised themselves with the final, agreed versions. The physical instruments were written in English but there was agreement between the enumerators, the local partner and the international team members on the Hausa interpretation of the questions. KIIs were conducted with the programme staff and with representatives of programme partners. FGDs were held with vaccine distributors and with VBVs. Propcom Mai-karfi assisted in facilitating interviews and focus discussions with stakeholders. The local partner conducted both KIIs and FGDs in Hausa and transcribed the meetings. Transcriptions were then translated into English and a report drafted focussing on each of the questions. The international members of the evaluation team reviewed the report and, where necessary, asked questions for clarity. The local partner then answered these questions directly or reverted to the respondents for clarity and reported back to the international team members. One FGD was held in the city of Kaduna with vaccine distributors. Three focus groups were held with VBVs in villages in Kaduna state. All of the respondents of these groups were men. KIIs were held with the owner of Agric-project-Concept International and with programme staff. Market specialist staff from another DFID funded Markets for the Poor (M4P) intervention, Market Development in the North of Ghana Independent Evaluation (MADE) Nigeria, were interviewed by Skype. This programme also has a poultry vaccination intervention in place and has cooperated with Propcom Mai-karfi. Propcom Mai-karfi progress reports and with quantitative endline data on the poultry vaccination intervention which the evaluation was collecting at the time, supplemented the primary data collected for this evaluation study. Additional qualitative data collected as part of the Women’s Economic Empowerment assessment, was also accessed to supplement this report. 6. Limitations to this Evaluation Resource limitations prevented the evaluation team from collecting data in all states participating in the Poultry Health intervention. However, at the time of conducting the focus group discussions the private sector partner, ACI, hosted a meeting of distributors of vaccinators from various states. The evaluation team were able to access this meeting and conduct FGDs and KIIs with individuals in this location. To some extent, this mitigates the limitation of geographic scope. However, this also limited the choice of respondents to those individuals who were attending the sponsored meeting. 4 Nippard, D. (et al) Adopt-Adapt-Expand-Respond: a framework for managing and measuring systemic change processes. Springfield Centre, March 2014 PO6311 Independent Evaluation of Propcom Mai-karfi – Draft Interim Phase 2018 Evaluation Report – April 2018 6 The evaluation team was not able to meet with all of the private sector providers mentioned in the report. One of the stakeholders is partner with a different M4P programme and it was felt that connecting with them might be detrimental to their current engagement. PO6311 Independent Evaluation of Propcom Mai-karfi – Draft Interim Phase 2018 Evaluation Report – April 2018 7 7. Findings The findings in this section are reported against the OECD DAC criteria, outlined below: • Relevance - The extent to which the activity is suited to the priorities and policies of the target group and recipient. • Effectiveness - A measure of the extent to which an activity attains its objectives. • Efficiency – For the purposes of this report, whether the intervention achieved its objectives on time and using cost-effective measures.5 • Sustainability – W hether the benefits of an activity are likely to continue after the close of the funded intervention. For the purposes of this evaluation, given the M4P focus of the programme, the report also looks at scale in this section. • Impact – The positive and negative changes produced by a development intervention 7.1 Relevance Key findings: The need for the vaccines was well established and well known before the programme ventured into this area. However, although a vaccine was available, a sustainable, business oriented solution providing access and vaccination services had not been designed or implemented. Propcom Mai-karfi identified several challenges within the market system and overcame these through building relationships with and between sector players. In so doing, the intervention has had an impact on these constraints. The underlying supply change challenges identified above were well recognised within the sector by commercial players before Propcom Mai-karfi identified them. It was also well known that rural households suffered the most losses as a result of these challenges. We have noticed this reoccurring problem for quite a long time where poultry farmers recorded mostly about 100% mortality rate as a result of the outbreak of Newcastle Disease. KII respondent, ACI. While the challenges were well-known, the value that Propcom Mai-karfi brought was identifying potential solutions, and presenting these solutions as business opportunities. In addition, the programme facilitated relationships between the stakeholders to implement these solutions. In terms of the intervention design, ACI reported that they looked at the intervention from a commercial point of view, but recognised that Propcom Mai-karfi had a programme objective of targeting women. To this end it was felt that an increased emphasis should be placed on identifying women as service providers. In the opinion of the respondents, this was only natural given that women are the primary carers of the household flocks. Women respondents from the Women Economic Empowerment (W EE) FGDs reiterated this point, [In] most cases …[care of the chickens is] the responsibility of the woman of the house who perhaps may even be the owner of the chickens…6 Woman FGD respondent, Bagwai, Kano State To overcome the lack of awareness and information of NCD and the vaccine, Propcom Project Manager (PM) explored a number of avenues. These included more traditional information routes such as radio adverts and working with state governments who had previously developed and implemented their own 5 The analysis in this section is limited to qualitative reports from data gathering from stakeholders. Cost effectiveness and overall programme efficiency are reported in the final evaluation report. 6 Emphasis added. PO6311 Independent Evaluation of Propcom Mai-karfi – Draft Interim Phase 2018 Evaluation Report – April 2018 8 publicity campaigns.7 The private sector partner also reported investing in promotional and awareness building material.8 Further to these PM explored the use of less traditional information channels, and targeted faith based organisations and CBOs, as touch points through which they could channel information and vaccines “ … the CBOs [Community Based Organisations] serve as peri-urban distributors, promoting ND-i2 [the vaccine] through their local networks … Many women are members of these CBOs hence ensuring that there is effective targeting of the marketing and sales activities”,9 and use these same organisations as sources to identify potential VBVs. Looking to overcome the lack of service providers and the challenge of the last mile distribution, the initial design of the vaccination intervention explicitly targeted women as VBVs. It was argued that this would allow the VBVs to overcome social and cultural restrictions allowing them access to households when men are at work. Several facts however, impacted on the success of this model; the lack of commercial incentive for these participants, who received free training and vaccines, quickly impacted the financial viability of the model. In the pilot over 100 people were trained, but very few set themselves up as vaccinators, instead opting to pursue white collar jobs.10 “In short the model then didn’t wear a business lens”. KII Respondent, ACI These shortcomings were overcome by following a screening process.11 The first step was to sensitise and advertise the process, then train the VBVs for a fee which included the cost of the first batch of vaccines. PM also facilitated a relationship between ACI and Notore, one of the programmes private sector partners in the fertiliser intervention, promoting the use of the fertiliser Village Promoters (VPs) as potential VBVs. This tie in has been met with mixed success, with ACI reporting excellent results in some states and less success in others.12 It is also felt that the same social and cultural restrictions the programme was seeking to overcome would restrict women from travelling from place to place building awareness and promoting the use of the vaccine. [The culture in the North] mostly discourages women from certain activities that will require them to move from one place to another. KII Respondent, ACI A business model redesign targeted any individuals who would be interested in becoming a VBV, and ACI reports that about 7% of the active VBVs with which they engage are women.13 Active male VBVs report that there are no difficulties with them entering a household during the day to vaccinate chickens, especially as the male head of the house might often have called to ask for the service on behalf of his wife.14 In terms of the relevance of the intervention, Propcom Mai-karfi identified constraints in the market system and worked with both private and public sector partners to overcome these constraints, with an aim of meeting the overall programme’s objectives, of alleviating poverty in the target population. 7 In 2011 the Plateau state government embarked on a self-funded publicity campaign to treat the disease, and subsequently approached Propcom Mai-karfi for assistance with a revised campaign. KII, Propcom Mai-karfi staff, May 2015. Similar initiatives were carried out in Jigawa state. 8 KII respondent, ACI, November 2018. 9 Propcom Mai-karfi, Annual Report, December 2016 10 Propcom Mai-karfi Self-Assessment Report, DFID Annual Review, January 2014, KII, Propcom Mai-karfi staff, May 2015. 11 Following the pilot Propcom Mai-karfi also attempted to profile the ideal VBV. See Propcom Mai-karfi Poultry Health Impact Assessment Report July 2014 12 KII respondent, ACI, November 2018. 13 KII respondent, ACI, November 2018. 14 FGD respondents. PO6311 Independent Evaluation of Propcom Mai-karfi – Draft Interim Phase 2018 Evaluation Report – April 2018 9 7.2 Effectiveness Key Findings: Vaccine production capacity constraints needed to be overcome to ensure a reliable supply of vaccines anticipated by the intervention, following effective awareness building campaigns. Importing vaccines from neighbouring Cameroon, while a viable short term solution, did not address domestic market system constraints. A direct intervention with the NVRI, commissioning already purchased equipment overcame this constraint. Partnering with a private sector partner and working with their existing distribution networks and supplementing these with a last mile model addressed this constraint. This model was supplemented with non-traditional distribution methods such as mobilising CBOs to work as distribution networks. There was some additional success in partnering with an existing network supported by the programme, focussing on fertiliser. The challenges that the intervention sought to overcome involved various stakeholders at multiple levels of the market system. A thermostable vaccine was already being produced by the NVRI but was marketed almost exclusively to commercial poultry farmers. Further to this, the Institute did not have the capacity or the mandate to develop distribution networks.15 After discussions with the programme the Institute agreed to increase their production of the vaccines to supply the intervention. However, challenges in this process relating to the need to replace old manufacturing equipment, affected the supply of the vaccine to the market place early in the intervention, threatening its ongoing roll out. The private sector partner reported that at that time the company received negative feedback from both its distributors, its vaccinators and its clients.16 [The] intervention has actually established high demand for the vaccine but distributors experienced serious difficulty in accessing the vaccine when the company suspended production during the first quarter of last year. That situation created a moment of pessimism for the distributors and the local poultry owners. Vaccine Distributor, Kaduna While the NVRI had purchased and installed a larger freeze dryer to increase their vaccine production capacity, the dryer had never been commissioned. “NVRI co-invested with [Propcom Mai-karfi] to engage the services of a team to commission their 30,000 vial capacity freeze dryer”.17 It appears that this has resolved the production challenges with the private sector partner; the retail distributors and the VBVs each reporting that they receive their requested stock when ordered.18 To overcome one weakness in the distribution network, the programme also worked with the NVRI to reinvigorate the Institute’s outstations19 to act as supply hubs for the thermo-sensitive vaccines, laying the groundwork for a widening of the potential distribution network. As reported above, a challenge that needed to be addressed was the lack of awareness of the disease and the need for the vaccine. While the need for the vaccine was never in doubt from a programme point of view, demand from rural households did not grow as quickly as anticipated, with the programme recognising that sensitising the population and building their awareness took considerable time, effort and resources.20 MADE Nigeria encountered similar challenges in promoting the vaccine in the Delta region.21 While the programme partnered with some state agencies in their campaigns, it also worked with the private sector partner and CBOs to develop more public facing communications and behaviour change material. These included posters, pamphlets and vests. Vaccine distributors and vaccinators 15 Propcom Mai-karfi Self-Assessment Report, DFID Annual Review, January 2014. 16 KII respondent, ACI, November 2017. 17 Propcom Mai-karfi, Annual Report, December 2016. 18 FGD respondents, Traders and VBVs, November 2017. 19 The programme also reports investing at least one of these outstations. Propcom Mai-karfi, Annual Report, 2017. 20 Propcom Mai-karfi Quarterly Report, March 2015. 21 KII respondent, MADE Nigeria, March 2018. PO6311 Independent Evaluation of Propcom Mai-karfi – Draft Interim Phase 2018 Evaluation Report – April 2018 10 report using this material to promote their services, although VBVs report not receiving a sufficient quantity and that the material needs replacement often.22 I … paste the banners in strategic locations such as mosques, churches and houses of village head, ward head and district head. Vaccine Distributor, Kaduna ACI reports that it has invested its own funds in the development and duplication of promotional and awareness raising material, as well as in training its own staff in awareness and promotion. ACI has also invested in the relationships with CBOs initially facilitated by Propcom Mai-karfi. It successfully utilised these relationships to build consumer awareness about the disease and the vaccine. 23 [A] significant proportion of vaccine sales came through the network of vaccinators established by CBOs. The CBO network allowed for rapidly increasing geographic scope and priming the market with very minimal resources. Following the CBO work, partners like ACI have been able to activate distributors in locations close to active vaccinators for sustained supply.24 However, traders and distributors report that the information is not as widespread as it could be, with more rural areas, (characterized by bad road and distances), not having high levels of awareness, and arguing that the public facing publicity material provided to themselves and the VBVs reached smaller numbers of people. They argued that the private sector partner should run media campaigns to increase awareness especially for those communities beyond the current reach of VBVs.25 In spite of this scepticism, reports from the quantitative survey indicate that awareness of NCD has increased not only within the target areas but also in areas where the vaccine is not yet available, indicating success of this strand of the intervention. At baseline, roughly two thirds of the total study population reported having heard of Newcastle Disease across the populations. This shifted to nearly all respondents having heard of it at endline; 100% of intervention, and 94% of comparison respondents. This shift in recognition suggests awareness of NCD has increased in both populations.26 Before partnering with ACI, Propcom Mai-karfi approached a number of private sector partners. ACI already distributed a range of veterinary and animal husbandry related inputs to a network of agro- dealers and paravets, and was positioned to assist in overcoming the distribution challenges. The company network was utilised in the distribution of the NCD vaccine, but needed widening to reach a broader range of beneficiaries. Through the programme ACI also worked with Notore’s fertiliser VPs and the CBOs identified by the programme. The pilot version targeted women as VBVs. The rationale for this was that as women the VBVs would have easier access to the households, especially during the day when male household members might be in the field. The programme deliberately identified women as VBVs to overcome anticipated social and cultural constraints. These identified individuals were trained and provided with vaccines which they were to use and then reimburse ACI for their cost then reorder. The model was not successful, with individuals not reimbursing the company for their initial stock. Subsequent changes to this business model included recruiting individuals, regardless of their sex, who were willing to invest by buying their initial stock of vaccinations, rather than the company providing these and expecting to recoup the outlay. 22 FGD respondents (VBVs) Kaduna, November 2017. 23 Propcom Mai-karfi Quarterly Report Sept 2015 24 Propcom Mai-karfi Annual Report, December 2017. 25 FGD respondents (Traders) Kaduna, November 2017. 26 Independent Evaluation for Promoting Pro-poor Opportunities in Commodities and Service Markets Mai-karfi Programme, Northern Nigeria Endline Evaluation Draft Report – Poultry Health, April 2018 PO6311 Independent Evaluation of Propcom Mai-karfi – Draft Interim Phase 2018 Evaluation Report – April 2018 11 While ACI reports being pleased with the current model from a business perspective,27 Propcom Mai- karfi recognises some limitations from a development and outreach perspective; The cost recovery sales strategy allowed partners [to] recruit individuals with entrepreneurial skills as vaccinators. However, there is a risk of creating an entry barrier for genuinely interested individuals who don’t have the funds to procure the start-up quantities of the vaccine.28 An assessment of the pilot led to a revision of the business model, and the screening of potential VBVs. The more successful VBVs from the pilot were those who were already involved in agricultural value chains; farmer, paravets, village promoters, veterinarians, etc. Students and other government workers were also part of this pilot model and were not effective as their colleagues.29 This finding is important as a forerunner of the development of the Community Animal Health Worker (CAHW) initiative. The last mile distribution appears to be working well in the targeted areas but faces a number of challenges in expanding into new areas. Many of the VBVs report challenges relating to increased transportation costs and inadequate equipment to transport and keep the vaccines at an appropriate temperature. Some of the VBVs report using local methods of clay pots and moist sand to keep the vaccines at a cooler temperature.30 One group of VBVs reported not being able to keep the vaccines at a suitable temperature, while another reported raising this issue with their supplier and being informed to use a standard cooler box. The effectiveness of the intervention addressed constraints at various points of the market system, including building awareness of the diseases and the vaccine, stimulating demand for the vaccine, ensuring that the vaccine was produced on time and made available in the relevant locations. Initial designs and models needed to be changed and adapted to ensure that the component part of the intervention continued to add value to the overall objective. 7.3 Efficiency Key findings: The private sector partners report investing their own funds in the intervention and report plans to invest more in the near future. Some partners have already replicated and adjusted model and are using this as a supplementary distribution model for their businesses. The programme has utilised its relationships with state and federal agencies to promote the intervention, and taking these actors concerns into consideration, has used the intervention as a foundational stone to develop a more expansive veterinary offering in rural areas. This report is not in a position to comment in the cost effectiveness of the poultry vaccination initiative as the programme does not keep financial records pertinent to each market intervention.31 However, catalysing private sector and additional investment in initiatives is a central strategy of M4P programmes, and this report comments on the level of investment leveraged by the programme at different points in the intervention. Propcom Mai-karfi invested in a number of areas of the poultry vaccination market system. The programme invested in the commissioning of equipment purchased and installed in the NVRI facility in Vos, in Plateau state. This process aided in increasing the production volume of vaccines. It is not clear at this time if the NVRI or the Federal Government will continue to invest in and maintain equipment for this and other vaccine production. However, recent developments centring on the NVRI’s access to its federal budget, appear to indicate financial support from the Federal Government to the Institute. The 27 KII respondent, ACI, November 2018. 28 Propcom Mai-karfi, Annual Report, December 2017. 29 Propcom Mai-karfi Poultry Health Impact Assessment Report July 2014 30 FGD respondents (VBVs). 31 Efficiency of the programme will be reported on holistically in the final evaluation report. PO6311 Independent Evaluation of Propcom Mai-karfi – Draft Interim Phase 2018 Evaluation Report – April 2018 12 programme attributes this support to its relationship with the NVRI and its advocacy with the Presidency,32 but this could not be confirmed by the evaluation. The programme has also invested resources and substantial amounts of time in advocacy with other veterinary bodies in Nigeria for a number of purposes; to initially advocate for previously untrained individuals to be recognised as VBVs with minimal training, and therefore be permitted to administer vaccines; for the subsequent development of a training curriculum for VBVs to be developed and rolled out nationally. This process was a foundational stone in the development of the CAHW programme. 33 Regarding investment in vaccine distribution, Propcom Mai-karfi, partnered with ACI, developing and refining the business model, VBV training and developing promotional and awareness building material. The programme reports that it is no longer directly investing in the initiative with ACI, and that the company is now in a position to invest in its own business model and expanded rollout.34 At the time of writing, ACI estimated that the company invested about 30% of the total costs and about 70% of the level of effort while the programme bore the bulk of the financial burden.35 The retail distributors and traders report that with the exception of the previously reported shortage of stock, vaccines are now readily available when they request them, often through a phone call, with some of the traders reporting that ACI delivers to their place of business. This appears to indicate that the distribution chain up to this level is working well. All but one of the traders buys the vaccine for cash or pre-pays for an order and then recoups the cost. One of the retail distributors has an account with ACI and purchases the vaccine order on account. The intervention has achieved results within the timeframe in which it has been running and the programme has leveraged additional private sector funding although the value of this additional funding from ACI is not confirmed. It is unlikely that the programme funds invested in the commissioning of the NVRI equipment were matched in kind. 7.4 Sustainability Key findings: Private sector partners report being pleased with the extent to which the intervention has generated additional income, although this vaccine related income is not seen as core to their business profitability. They view the intervention as an opportunity to expand their distribution network, to identify more clients and to cross-sell additional veterinary products to this new market. Crowding in of different private sector partners, and their replication of the model also indicates sustainability of the intervention. As a stand-alone product the vaccine does not provide sufficient margin to warrant an individual vaccinator’s income, but can be a contributory income stream of a multifaceted product and service offering. The VBVs view their vaccination offering as one income stream that contributes to a percentage to their personal income, and in some cases VBVs have multiplied this by developing their own distribution and vaccination networks. The increasing and ongoing demand for the vaccine from the households indicates a sustainability of demand, as long as the NVRI can continue to manufacture the product. The replication of the model in different formats and the expansion of the marketing of the vaccine by another developmental organisation echoes the intervention’s sustainability. In terms of building the sustainability of an initiative it is important that the intervention is 1) sufficiently profitable for the private sector business, 2) that the business can achieve sufficient scale to improve 32 Propcom Mai-karfi Annual Report 2016 33 Propcom Mai-karfi Annual Report 2016. 34 Propcom Mai-karfi Quarterly Report March 2016. 35 KII respondent, ACI, November 2017. These figures are in contrast to those reported earlier by the programme where it was estimated that “for every naira invested by [the programme], ACI will match it … 2.5fold”. Propcom Mai-karfi Annual Report 2015. PO6311 Independent Evaluation of Propcom Mai-karfi – Draft Interim Phase 2018 Evaluation Report – April 2018 13 the profitability and 3) that there are no significant challenges confronting those rolling out the initiative. In addition, from an M4P perspective, the concept of scale introduces “crowding in” or replication of the model by other market system stakeholders. This section will examine these issues at each level of the vaccination market system. It is not clear from either programme reports or from the primary data gathered in this evaluation to what extent the NVRI has developed plans that reflect ongoing duplication of vaccines. Propcom Mai-karfi reports that the Presidency has been informed of the significant economic impact of halting the vaccine production and a resultant outbreak of NCD. Nevertheless, the stop in production reported above led to the programme facilitating ACI importing the vaccine from neighbouring Cameroon.36 This suggests that the distribution of the vaccine will be sustained regardless of its production within Nigeria. Private sector partners can import vaccines, should the need arise. ACI reports that they wish to continue with the distribution of the vaccine and the identification and training of VBVs. They see this as a viable business opportunity, and are generally pleased with the extent to which the market has grown. In spite of this growth, the vaccine intervention is not central to the ACI business, who perceive it as a minor offering in its stable reporting that [the] intervention’s contribution to the company’s share of its gross profit was less than 1%, which is insignificant.37 However, the company sees the vaccine offering as an opportunity to provide other goods and services to new clients which it otherwise would not have had.38 The company reports that it is in a financial position to continue to offer the service without the financial investment of the programme ,and that it is pleased with the extent to which the intervention has grown, reporting significant growth over the intervention period and attributing this to increased awareness and the effectiveness of the vaccine. there was a time we supplied more than 3,000 vials of the vaccines and were sold within a day [to] the major dealers....39 Importantly, MADE Nigeria, another M4P programme, also piloted the ACI / Propcom Mai-karfi vaccine business model. This programme, operational in the Niger Delta, made use of both the curriculum from Propcom Ma-karfi and initially developed a relationship with ACI for the pilot.40 This pilot found that offering the vaccine on its own as a service, did not generate sufficient income for vaccinators requiring high levels of work and support to roll out. The model that they were duplicating at the time was that of offering the vaccine as a stand-alone input or service and they have subsequently revised the model with their vaccinators now offering a range of related goods and service. The reason the pilot was not successful was because of the attrition rate [of the vaccinators]. The money was not enough. MADE Nigeria respondent 36 Propcom Mai-karfi Annual Report 2016. 37 KII respondent, ACI, November 2017 38 KII respondent, ACI, November 2017. Newcastle disease is not mentioned and the ND-I2 vaccine is not listed on the ACI website as a product, nor is any mention of VBV training, or linkages and partnerships with CBOs. http://agriprojectsinternational.com/ Accessed March 2018. 39 KII respondent, ACI, November 2017 40 The relationship between Propcom Mai-karfi and MADE Nigeria goes beyond one programme making use of the other’s outputs and includes cooperation at trade shows (Propcom Mai-karfi Quarterly Report September 2015) and co-ordinating approaches in dealing with the NVRI and VCN (Annual Review 2017, Annual Report 2017). It is worth noting however, that the relationship between the programmes needed to be nurtured and developed and that early offerings by MADE could have derailed the Propcom Mai-karfi poultry vaccine intervention (Propcom Mai-karfi Quarterly Report June 2015). PO6311 Independent Evaluation of Propcom Mai-karfi – Draft Interim Phase 2018 Evaluation Report – April 2018 14 MADE Nigeria, however, learned from these lessons and made adaptations to the model, partnering with other veterinary pharmaceutical companies. They developed a model to roll out a service to their own beneficiaries where service providers could offer a range of veterinary services to household in partnership with Turner Wright and Zygosis.41 MADE Nigeria also reported that ACI did not have the organisational capacity to provide the service so far from its home base in Kaduna. It has subsequently developed relationships with other argi-product and veterinary suppliers and is successfully rolling out a variation of the business model in the states in the Niger Delta region. The concern expressed by MADE Nigeria of the capability of ACI is reflected in Propcom Mai-karfi reports and was reflected again in this evaluation’s interview with the respondent. The programme reports reflect on ongoing relationship with ACI where capacity of the company is lacking and this mirrors their inability (or unwillingness) to grasp market opportunities.42 Similarly, in the KII for this evaluation, the company respondent repeatedly reflected on the lack of government action as the reason for limited progress or growth in the poultry industry.43 This view appears to reflect little of the catalytic thinking that the M4P model promotes – the value of market development and innovation as a model to alleviate poverty – in spite of Propcom Mai-karfi’s substantial capacity building investment in the company. The programme has not limited its vaccine related outreach to this one private sector partner and is developing relationship with other companies.44 The retailers and traders see the vaccine as central to their overall offering. The vaccine contributes varying percentages to the profitability of the businesses45 but all of them reported that it is sufficiently important for them to want to continue in its distribution and application. One of the retailers reported providing or encouraging the provision of free vaccination services to outlying rural areas as part of his Community Development Service, reporting that this happened in Kafancha in 2016.46 The distributors have also started replicating their own VBV networks increasing their outreach and improving the scale of their offering,47 Some of these additional partners are adopting the proven business model and adapting it to their own specific needs, reflecting the M4P concept of “crowding in”. At distributor level, there are modest reports of copying of the cost recovery sales strategy for recruiting and capacity building of vaccinators. This strategy, developed in 2015 to help companies efficiently recruit and link vaccinators to their distributors is gradually catching on. Greenflash, a distributor for ACI in Makurdi successfully used the model to expand their … network. These distributors aside, early in the intervention, other companies reported being concerned about their reputation with the regulatory authorities (specifically the Veterinary Council Nigeria (VCN)) regarding their engagement with “unlicensed” vaccinators. The programme’s relationship with the VCN and this institution’s subsequent endorsement and support of the vaccination intervention has laid the groundwork for a wider network of VBVs who might also perform additional work as CAHW.48 The VBV 41 KII MADE Nigeria respondent, February 2018. 42 See for example Propcom Mai-karfi Quarterly Report September 2015. 43 KII respondent, ACI, November 2017. 44 For example, the programme reports developing relationships with Agrited, AnimalCare and Thlalakolo, Propcom Mai-karfi Annual Report 2015 45 When asked to report on the importance of the vaccine to their income in terms of a percentage, traders reported percentages varying from 1% to 55%. FGD respondents (Traders) Kaduna, November 2017. 46 FGD respondents (Traders) Kaduna, November 2017. 47 Propcom Mai-karfi Annual Report 2016. 48 It is expected that the CAHW initiative will train rural based individuals to offer limited veterinary services to farmers in their communities, overcoming market system constraints of limited capacity and distance and improving animal husbandry in remote areas. “Trained vaccinators can be potential Community-based Animal Health Workers … farmers trust them and come to them for advice for all their animal health needs. Existing network of vaccinators can therefore be the first ‘go to’ people to promote the CAHW programme.” Propcom Mai-karfi Annual Report 2017. PO6311 Independent Evaluation of Propcom Mai-karfi – Draft Interim Phase 2018 Evaluation Report – April 2018 15 initiative and the programme’s facilitation process with both private sector companies and regulatory authorities might be seen as culminating in the CAHW proposal.49 At a VBV level, vaccinators report that there is a significant demand for repeat business for their services and that there are high levels of customer satisfaction. Some of the VBVs report clients calling them to administer vaccines on new born chicks. They also report that neighbours and friends have requested training so that they too can administer vaccines. The programme reported a similar development replication in 2015 in Gombe and Jigawa50, observing Organic growth at Village Based Inoculator (VBI) level, as VBIs is on their own, creating downlines and [evolve] to some subtype of distributorship. All of the VBVs report profiting from their vaccination practice and although none of them are dependent on this as a sole source of income, they agree that vaccination is a regular source of income, and that they envisage themselves continuing to provide the service. Reported income from vaccination as a percentage of their total income varies from 10% to 25%. All of the VBVs interviewed expressed a view that they would be able to reach more outlying districts if provided with a means of transportation, and suggested that they be given the opportunity to buy motorcycles on credit.51 The intervention has created an awareness of NCD and a demand for the vaccine amongst the households it aimed to impact. These two factors have created a demand-led pull for the vaccine and related services, with private sector companies expanding their offering, replicating and adapting the business model. In addition, another donor programme has learned from this process and is partnering with two additional private sector companies in another area of the country, increasing the sustainability and scale of the initial offering. 7.5 Impact Key findings: The intervention has had impact at various points in the vaccine market system; from manufacture, to distribution, to application. In terms of programme goals, there has also been impact on household poverty. Importantly the spread of information regarding NCD and its treatment has made households more aware of the disease and preventative measures. This improved access to information has also lead to an almost all encompassing practice of poultry vaccination in intervention areas. As a result, the household has healthier chickens, the sale of which generates greater income for the household. The income has also been used to for household purposes as well as reinvesting it in household flocks and in other potential income streams. It is likely that these purchases aid in building household resilience. The intervention was designed to increase the household uptake of poultry vaccines amongst the rural poor in northern Nigeria. The independent quantitative endline survey52 conducted as part of this broader evaluation found that the uptake of vaccines within the intervention area was nearly 100%. This behaviour change has resulted in decreased poultry mortality and a resultant increase in poultry related income amongst households which have vaccinated. The intervention has assisted beneficiaries, many of whom are women by helping them meet “…emergency needs, improved their 49 “Two veterinary pharmaceutical companies, Turner Wright and FDH, are in partnership negotiation process to work with CAHWs; one distributor in Kaduna, Ambuvet, already mentoring 6 prospective CAHWs.” Propcom Mai-karfi Quarterly Report September 2017 50 Propcom Mai-karfi Annual Report 2015. 51 FGD respondents (VBVs). 52 Independent Evaluation for Promoting Pro-poor Opportunities in Commodities and Service Markets (Propcom) Mai-karfi Programme, Northern Nigeria Endline Evaluation Draft Report – Poultry Health, April 2018 PO6311 Independent Evaluation of Propcom Mai-karfi – Draft Interim Phase 2018 Evaluation Report – April 2018 16 income, help people to cope with seasonal shocks of feeding [increasing household] resilience”. This is reinforced by the independent end line evaluation of the poultry vaccination intervention where, “…income available to the intervention women … is unavailable to the [farmers in the] comparison population”. 53 FGD respondents in the independent evaluation WEE study54 reported that as a result of the application of the vaccine, opportunities for their girl children had improved, with the girls caring for chickens as their own economic assets, using the birds mainly in case of emergencies or in case of special occasions (getting into secondary school and weddings). The same respondents reported that other women in their community who had not had access to the vaccination services (and additional advice from the vaccinator) have approached the respondents for information regarding poultry husbandry, demonstrating the relevance of the intervention. Respondents in the endline study also report investing in more chickens and using poultry related income to diversity their income streams.55 This qualitative study reports similar results at a household level with VBVs reporting clients having larger flocks, earning more from healthier birds, and clients diversifying their income streams A farmer called me … and she’s happy to tell me that she has about 45 full grown chickens now, and when she sells them she intends to buy a grinding machine. FGD respondent, VBVs VBVs report greater success in terms of roll out and impact than they had been anticipating. All of the VBVs reported that their clients were satisfied with the results and a number of them reported receiving unsolicited gifts from their clients as a result of providing the service.56 Impact on the VBVs is an increase in demand for a variety of services from them, with VBVs reporting households asking for assistance a wide range of veterinary cases from poultry to goats and sheep, reinforcing the need for the CAHW initiative. VBVs report: We sometimes assist them in handling minor cases within our capacity and report major cases to our major distributors who are Veterinary doctor for further advice or treatment. I have also started expanding this service to meet up the current demand of the local poultry farmers by providing drugs for their ruminants. FGD respondents, VBVs This diversified demand for services indicates a need for low cost accessible services within the rural areas. While the VBVs currently report that vaccination services contribute a relatively small proportion of their income, a demand for a range of services might increase this proportion substantially. The anticipated impact of the programme; to alleviate poverty through increased poultry related incomes, has been addressed through this intervention. Respondents who have benefitted form the intervention report increased sizes of their flocks, improved income from selling chickens plans to increase their size of their flocks. In addition, some respondents report diversified income streams from investing poultry related income in assets and businesses unrelated to poultry, increasing household resilience. 53 Independent Evaluation for Promoting Pro-poor Opportunities in Commodities and Service Markets (Propcom) Mai-karfi Programme, Northern Nigeria, Endline Evaluation Draft Report – Poultry Health April 2018. 54 Independent Evaluation for Promoting Pro-poor Opportunities in Commodities and Service Markets (Propcom) Mai-karfi Programme, Northern Nigeria; Effectiveness in Reaching Women and Women’s Economic Empowerment, April 2018 55 Independent Evaluation for Promoting Pro-poor Opportunities in Commodities and Service Markets (Propcom) Mai-karfi Programme, Northern Nigeria, Endline Evaluation Draft Report – Poultry Health April 2018. 56 FGD respondents (VBVs) Kaduna, November 2017. PO6311 Independent Evaluation of Propcom Mai-karfi – Draft Interim Phase 2018 Evaluation Report – April 2018 17 8. Conclusions The intervention aimed to overcome several constraints in the vaccine market system, as outlined in Section 2 and illustrated in Figure 1: Vaccine Value Chain Showing Market System Challenge Areas. It aimed to do so in a way that promoted market oriented, sustainable response. This section summarises the findings of the evaluation and, in keeping with an M4P lens in the methodology in Section 5, applies the AAER lens to determine the extent of systemic change within the vaccination market system. As an input into the final programme evaluation, this evaluation report also attempted to answer key evaluation questions, identified in the programme inception report. In its initial planning, the programme identified poultry as a low barrier opportunity to address household poverty in Northern Nigeria. The programme also identified market system constraints at various points and designed the intervention to overcome these constraints. This meant working with regulatory stakeholders, and addressing their concerns regarding, amongst other things, the proliferation of unlicensed and untrained animal health care workers. The intervention also addressed the production and distribution constraints within the market system, piloting and refining a business model to distribute into more rural areas. In addressing the constraint of information access and increasing knowledge of NCD and the vaccine, the programme worked with state agencies which were already informing audiences, and through their private sector partners, worked with the last mile distributors equipping them with information and promotional material. The forthcoming endline quantitative assessment speaks to the relevance of the impact, with beneficiaries reporting in increased flocks, healthier chickens, and more income from these chickens. The programme has actively shared the learnings of this intervention, both with other private sectors players allowing for them to crowd in and replicate and adapt the model for their own purposes from the Propcom, and with other donor programmes. In the latter case, MADE Nigeria reported from working Mai-karfi base, valuing their learnings and making adaptations suitable for their own programme purposes. A number of sector companies have taken ownership of the business model are making adaptations, and are using these models to reach into the more rural market. These adaptations, together with the CAHW initiative, has opened an opportunity for more affordable veterinary services in rural areas – a primary constraint that the programme sought to overcome. In this sense, the intervention has been catalytic. It is too early to tell if the programme’s interventions with the regulatory and state authorities can be similarly described. Table 1: AAER Summary of the Intervention summarises the intervention’s results, in terms of systemic change in the market. Table 1: AAER Summary of the Intervention Adapt Respond • Distributors and CBOs identify and train VBVs to act as service providers and agents. • VBVs train others in vaccination, forming their own, smaller distribution networks. • ACI reports considering expanding the geographic scope of the current outreach on its own and of adding additional veterinary products and services. • The intervention has been foundational to the formulation and development of the CAHW initiative. • The private sector partners of MADE Nigeria are reported as playing an initial role in the national CAHW initiative. Adopt Expand PO6311 Independent Evaluation of Propcom Mai-karfi – Draft Interim Phase 2018 Evaluation Report – April 2018 18 • ACI piloted an intervention in partnership with the programme. • ACI worked with the initial model and made changes based on the pilot results. • Regulatory authorities partnered with the programme to promote NCD awareness. • VCN endorses and promotes the formulation of a national vaccination curriculum • Additional private sector partners use the business model and make changes suitable for their own purposes. • MADE Nigeria learns from Propcom Mai-karfi and makes changes to the model, identifying and partnering with other private sector partners in the Niger Delta region. The intervention attempted to meet women’s by equipping women as VBVs and to be active players in the market system. However, cultural constraints impeding women’s travel from area to area made this opportunity less than ideal. In revising the model, women reported no concerns about asking a man to perform vaccinations, as long as this the male head of the household has approved or facilitated the visit. Women are the main caretakers of the household flock, and are often owners of chickens as well. As such they benefitted either directly or indirectly from this intervention. Beneficiaries have reported increased incomes from sales of chickens. This income is reported as being used to increase the size of household flocks, to purchase additional income generating equipment, to hire or buy additional land for farming or to pay for education and training. In all of these cases, the opportunity exists for the resilience of the household to be improved through diversified income or improved opportunities as a result of improved education.