Copperbelt Livelihoods Project Effectiveness Review – Full Report Women’s Empowerment and Livelihood Support Oxfam GB Women’s Empowerment Global Outcome Indicator December, 2011 FINAL Photo: Emma Walsh Copperbelt Livelihoods Project Effectiveness Review Table of Contents Table of Contents ......................................................................................................................... 2 1. 2. 3. Introduction and Purpose ..................................................................................................... 4 The Copperbelt Livelihoods Project ....................................................................................... 4 Intervention logic of the support provided to targeted households ........................................ 5 4. Women’s Empowerment outcome indicator, other measures used, and their conceptual underpinnings.............................................................................................................................. 7 4.1 The Women’s Empowerment Indicator ................................................................................... 7 4.2 Other outcome measures ........................................................................................................ 8 4.3 Measuring Intervention Exposure.......................................................................................... 10 5. Impact Assessment Design .................................................................................................. 10 5.1 Limitations in Pursuing the Gold Standard ............................................................................ 10 5.2 Alternative Evaluation Design Pursued .................................................................................. 11 5.3 The Comparison Group .......................................................................................................... 12 6. Methods of Data Collection and Analysis............................................................................. 13 6.1 Data Collection ....................................................................................................................... 13 6.2 Data Analysis .......................................................................................................................... 13 6.3 Main Problems and Constraints Encountered ....................................................................... 14 7. Results ............................................................................................................................... 14 7.1 General Characteristics .......................................................................................................... 14 7.2 Intervention Exposure ............................................................................................................ 15 7.3 Global Outcome Indicator for Women’s Empowerment ....................................................... 16 7.4 Household decision-making ................................................................................................... 18 7.5 Perceived ability to influence affairs at the community level ............................................... 19 7.6 Women’s self-efficacy ............................................................................................................ 21 7.7 Female asset ownership......................................................................................................... 22 7.8 Household asset ownership ................................................................................................... 24 7.9 Production of maize ............................................................................................................... 26 7.10 Production of groundnuts .................................................................................................... 28 7.11 Income earned from selling vegetables ............................................................................... 29 8. Conclusion and Programme Learning Considerations ........................................................... 30 8.1 Conclusions ............................................................................................................................ 30 8.2 Programme Learning Considerations ..................................................................................... 31 2 Copperbelt Livelihoods Project Effectiveness Review Executive Summary As part of Oxfam Great Britain’s (OGB) Global Performance Framework (GPF), mature projects are randomly selected each year and their effectiveness rigorously assessed. Zambia’s Copperbelt Livelihoods Project was selected in this way under the women’s empowerment thematic area. One of the key areas assessed through this process was the extent it has promoted change in relation to OGB’s global indicator for women’s empowerment: % of supported women meaningfully involved in household decision making and influencing affairs at community level The reviewed project was implemented by a local partner, the Sustainable Agriculture Programme (SAP), where it targeted 1,000 small-scale farmers (of whom 60% are women) living in ten villages in Kitwe district of Zambia’s Copperbelt Province. In addition to empowering women, the project sought to bolster household income and food security and reduce vulnerability through the provision of agricultural inputs and increasing market access. In October 2011, with the support of an external consultant, a household survey was administered to 173 randomly selected women from the 10 intervention villages, as well as 248 comparison women from neighbouring communities. The survey comprised of questions not only relevant to the global indicator but also the project’s other intended outcomes. In order to compare ‘like with like’, statistical analysis of the resulting data was undertaken using propensity score matching (PSM) and multivariable regression (MVR) to control for observable differences between the intervention and comparison women. Overall, no statistically significant difference was identified between the intervention and comparison women for the global women’s empowerment indicator. However, when the two constituent dimensions of this indicator are decomposed, a different picture is revealed. In particular, a statistically significant difference exists in favour of the intervention women in relation to their perceived ability to influence how their communities are governed, but with no such difference for their reported involvement in household level decision-making. A mixed picture also emerges for other aspects of women’s empowerment. In particular, the women supported by the project are more likely to own one or more ‘strategic’ asset but, at the same time, demonstrate less self-efficacy. Given that the reviewed project was also focused on bolstering household income and food security, it is also of interest to explore whether it successfully did so. Towards this end, data were collected on household asset ownership and crop production and sales. A statistically significant difference in favour of the intervention women was only consistently identified for the former measure. While there is evidence that the project has brought about positive change, there are a number of ways in which it can be strengthened. These include: Review options for making women’s empowerment interventions more explicit and tangible in the project’s design. Consider increasing efforts to further bolster agricultural production and support the marketing of agricultural commodities. Explore options to increase the effectiveness and impact of the project further by accompanying direct implementation with an advocacy strategy to make relevant policy and institutional changes Follow up on some of the specific findings from this report with further qualitative research 3 Copperbelt Livelihoods Project Effectiveness Review 1. Introduction and Purpose This report documents the findings of an effectiveness review of a livelihoods project that sought to intentionally empower women. Oxfam GB has put in place a Global Performance Framework (GPF) as part of its effort to better understand and communicate its effectiveness and enhance learning across the organisation. This framework requires programme/project teams to annually report generic output data across six thematic indicator areas. In addition, modest samples of mature projects (e.g. those closing during a given financial year) under each thematic indicator area are being randomly selected each year and rigorously evaluated. One key focus is on the extent they have promoted change in relation to relevant OGB global outcome indicators. The following global outcome indicator was endorsed for the women’s empowerment thematic area: % of supported women meaningfully involved in household decision-making and influencing affairs at community and enterprise level The conceptual underpinnings of this indicator are presented in Section 3.0 below. The work that took place in the Copperbelt province of Zambia in October 2011 was part of an effort to capture data on this indicator. This report presents the findings from the evaluation process, where a specially designed questionnaire was administered to women supported by the Copperbelt Livelihoods Project and similar women residing in neighbouring areas that were not. However, before doing so, Section 2 first briefly provides background information on the project and the context in which the support is being provided, while Section 3 explains the project’s intervention logic. Section 4, Section 5, and Section 6 follow by presenting the indicators used, the impact evaluation design pursued, and the methods of data collection and analysis, respectively. Section 7 is the longest section of this document. Its subsections include those related to basic descriptive statistics, intervention exposure, and finally the overall differences between targeted and comparison women. Section 8 concludes the document with general conclusions and suggestions for strengthening women’s empowerment and livelihoods support in the Copperbelt province. 2. The Copperbelt Livelihoods Project The Copperbelt Livelihoods Project was implemented by a local partner organisation, the Sustainable Agriculture Programme (SAP). It sought to increase household income and food security, and reduce vulnerability among approximately 1,000 small-scale farmers (of whom 60% are women) living in 10 villages in Kitwe district, Copperbelt Province, Zambia. This was primarily through the provision of agricultural inputs and increasing access to markets. The project’s specific activities include: Provision of quality seeds and fertilisers Support Agricultural Extension Officers to provide advice and technical support Support the training and construction of improved grain storage facilities Support the development of market gardens during the dry season, including provision of inputs and training 4 Copperbelt Livelihoods Project Effectiveness Review The project employed a range of interventions aimed at improving livelihoods and strengthening the role of women. Training in gender issues, such as greater recognition of women’s roles in the household/community, and encouraging leadership opportunities for women in communities. Training in health and HIV/AIDS issues Training in advocacy and lobbying The context in which the project was implemented is plagued by chronic poverty, poor production, low market access, climatic related shocks, and a high prevalence of HIV and AIDS. Figure 2.1: Location of Kitwe District, Zambia 3. Intervention logic of the support provided to targeted households The following diagrams illustrate simple ‘theories of change’ for how the project’s key interventions sought to increase household income and food security, as well as promote greater decision-making and empowerment for women. The main focus of this report relates to the latter. However, data were also collected on household asset ownership, thereby, enabling an assessment of the performance of the project against this measure. 5 Copperbelt Livelihoods Project Effectiveness Review Key project interventions included the provision of fertiliser and seed, together with training in farming methods. Oxfam GB, in partnership with SAP, has been supporting 10 communities in Kitwe district since 2009, through the Copperbelt Livelihoods Project. One of the key interventions has been the provision of agricultural inputs and related support to approximately 1,000 households (Figure 3.1). Farming families have been directly supported with fertiliser and seed input packages (maize and groundnuts) for the 2009/10 and 2010/11 seasons. Furthermore vegetable seeds (cabbage, rape, onion, tomatoes, and okra) were distributed for winter vegetable production during 2010. These inputs were supported by agricultural extension officers who provided training in agricultural methods. The aim of these interventions is to improve all-year round production of crops, thereby contributing to increased agricultural profits and household income. Households were also trained in the construction of improved household grain storage bins (Figure 3.2). A smaller number were provided with materials to construct demonstration bins. The underlying logic is that surplus grain is stored following harvest, which can either supplement the next harvest, thereby reducing expenditure on new seed, or can be sold for greater profit when market prices are high. This is to then increase agricultural profit and, in turn, household income and improved food security. Figure 3.1 Intervention Logic: Agricultural inputs and support Improved Income & Food Security Increased Household Income Increase Agricultural Profits Less Expenditure on Seed and inputs Increased production of crops Provision of seeds and fertiliser Ag extension officer support Figure 3.2 2 Intervention Logic: Household grain storage bins Improved Income & Food Security Increased Household Income Increase Agricultural Profits Better Access to Grain in “Hungry Season” Less Expenditure on Seed Improved Grain Storage Improved household grain storage bins In delivering these inputs to women in approximately 60% of cases, the aim is that these interventions is intended to also contribute to women’s control over strategic assets and involvement in decision making at a household and community level. 6 Copperbelt Livelihoods Project Effectiveness Review Training community members on gender issues was also a component of the project. The project also provided training on gender issues. This included explaining the importance of women’s roles in the household and community, highlighting the need for recognition of additional opportunities for women to be involved in community leadership. The training was delivered separately in each of the 10 villages, and field workers were available to follow up with households if required. The desired outcome from this intervention is greater involvement of women in household and community decision making, as well as increased ownership and control over assets by women. These were to lead then to greater decision making and general empowerment for women. Figure 3.3 2 Intervention Logic: Gender Issues Training Greater decisionmaking and empowerment for women Increased ownership and control over “strategic” assets Greater involvement of women in household decision making Greater involvement of women in community decision making Improved recognition of women’s role in household & community affairs Training in gender issues The project included other interventions, which are outside the scope of this assessment, including training of community leaders in advocacy and lobbying skills to engage government and civic leaders. 4. Women’s Empowerment outcome indicator, other measures used, and their conceptual underpinnings 4.1 The Women’s Empowerment Indicator OGB’s women’s empowerment indicator comprises of two components or subindicators. The first examines both the breadth and depth of women’s involvement in household decision-making. Breadth is defined in terms of the number of decision-making areas in which women are involved, e.g. decisions around food preparation, personal travel, and family planning. The instrument comprises of 25 decision-making areas. Depth, on the other hand, is defined in relation to the extent of involvement in each area, i.e. exclusive involvement, very strong involvement, joint involvement, some involvement, or no involvement. Not relevant is also an option if the decision-making area does not apply to the respondent’s household, e.g. decision making areas that pertain to children when none reside in the household. Women are also asked a follow-up question when they express they either have only some involvement or no involvement in a particular decision-making area: To what extent do you think you could be more involved in making decisions in this area if you really wanted to? If women respond with not at all or only to a small extent, it is assumed that women are barred from making decisions in that particular area. 7 Copperbelt Livelihoods Project Effectiveness Review The global indicator seeks to measure both women’s involvement in household decision-making and their ability to influence how their communities are governed. However, if they respond by stating that that they could be more involved if they wanted to, either to a medium or high extent, this is taken to mean that they would have the freedom to become more involved, but have chosen not to for one reason or another. In other words, the follow-up question is included in the questionnaire given the fact that some women may choose not to participate in particular household decision-making areas. What is of particular interest is the extent that women have the potential of being involved in the various decision-making areas. As such, based on their responses to the follow-up question, the women’s decisionmaking score for the area in question is adjusted accordingly, e.g. kept the same if the women reports that she could not be involved in such decisions or is adjusted upwards if she reports that she could be more involved. The second component of the indicator examines the extent to which women perceive they are able to influence the running of affairs at the community level, e.g. influencing the decisions of community leaders or obtaining leadership positions if they really wanted them. This is done by using a ten-item six-point Likert scale. Female respondents, in particular, are asked the extent they agree or disagree with the following statements: 1. You would be in a position to change things in your community if you really wanted to. 2. It would be extremely difficult for you to obtain an important leadership position in your community even if you really wanted one. 3. Despite trying really hard, it would be very difficult for you to influence how leaders are chosen in your community. 4. It would be quite easy for you to influence many of the decisions made by most of the leaders in your community if you felt it important to do so. 5. Community-level decisions you feel are important would very difficult for you to influence. 6. There are real opportunities open to you to participate meaningfully in important decision-making bodies in your community. 7. Women in your position could never be influential people in your community; the barriers are just too big. 8. If local leaders were doing things you did not agree with, you would just have to adapt and could not do much to stop them. 9. Things have really changed in your community; there are now many opportunities for women in your position to become influential actors in how your community is governed. 10.There are many initiatives happening in your community where your voice could never be heard in any meaningful way. 4.2 Other outcome measures As reviewed in Section 3 above, the support provided to the targeted households is intended to bring about a number of other outcomes, in addition to empowering women. Given this, data were collected on a number of other outcome measures. These include those relating to household ownership of assets, agricultural production, women’s access to and control over strategic assets, and women’s selfefficacy. 8 Copperbelt Livelihoods Project Effectiveness Review Data were also obtained on household asset ownership to measure household wealth status. Household ownership of assets Household ownership of assets is a recognised way of measuring established household wealth status. Households were asked whether they own a number of different assets from lists of various types, e.g. radios, bicycles, and livestock, as well as the materials used to construct the roof, walls, and floors of their homes and size of their agricultural land holdings. It is assumed that ownership of such assets in the past is something that can be reliably recalled. Respondents were therefore also asked to recall this information with respect to the baseline period, thereby, enabling the reconstruction of baseline data for this particular variable. A statistical method known as principal component analysis (PCA) was run on all the assets in the dataset separately for both periods to develop asset indices. This is a data reduction method that transforms a number of possibly correlated variables into a smaller number of uncorrelated variables called principal components. In more simple terms, it narrows in on those assets where there is significant variability in ownership and uses them to assign scores based on the possession of these assets. Hence, households that possess more and less of these assets obtain higher and lower scores, respectively. The first principle component, in particular, accounts for as much variability in the data as possible, and forms the basis of the asset index. The resulting index, itself a continuous measure, can be divided into quantiles (e.g. three groups) to define different wealth groups (e.g. the poor, middle, and rich). Agricultural production Respondents were asked about their production of crops, including maize and groundnuts, in the last 12 months compared with 2008 (prior to project intervention). Respondents were similarly asked about income earned from selling vegetables. While the potential for recall bias is obviously considerably higher than in the case of household asset ownership, comparing the difference in production/income earned over the last 12 months and 2008 enables the magnitude of change in production/income earned between the intervention and comparison groups to be estimated. Women’s ownership of strategic assets To measure women’s access to and control over strategic assets, the female respondent is first asked whether she either fully or jointly owns any of the following: land, livestock, the home/dwelling she lives in, any other house or building, any major farm assets, any off-farm income generating assets, or any other valuable asset such a TV, jewellery, or furniture, as well as whether they have any savings. For each asset she owns fully, the respondent is then asked whether she would have the final say in decisions pertaining to its sale if this was desired or necessary. An affirmative response to this question is assumed to indicate that the respondent has full ownership of the asset. Women’s self efficacy Finally, women’s self-efficacy was measured using an adapted version of the General Self-Efficacy Scale (GSE). This is a four-point Likert scale that asks respondents the extent to which each of the following statements is true for them: 1. You can always manage to solve difficult problems if you try hard enough. 2. If someone opposes you, you can find the means and ways to get what you want. 3. It is easy for you to stick to your aims and accomplish your goals. 4. You are confident that you could deal efficiently with unexpected events. 9 Copperbelt Livelihoods Project Effectiveness Review 5. Thanks to your resourcefulness, you know how to handle unforeseen situations. 6. You can solve most problems if you invest the necessary effort. 7. You can remain calm when facing difficulties because you can rely on you coping abilities. 8. When you are confronted with a problem, you can usually find several solutions. 4.3 Measuring Intervention Exposure There was a desire to also assess the extent to which the intervention and comparison women were exposed to the types of support targeted at the households. As such, the respondents were asked the extent they or any other members of their households had received agricultural inputs or training or support from an agricultural extension worker. As the project also provided training on gender issues and advocacy, specific questions were asked about exposure to these interventions. For each support measure, whether inputs or training, respondents were asked either how many times it had been received or the number of household members trained since 2008. 5. Impact Assessment Design 5.1 Limitations in Pursuing the Gold Standard The aim of the effectiveness review was to estimate what would have happened to the supported women had they never been supported. The core challenge of a social impact evaluation is to credibly estimate the net effect of an intervention or programme on its participants. An intervention’s net effect is typically defined as the average gain participants realise in outcome (e.g. income) from their participation. In other words: Impact = average post-programme outcome of participants – what the average post-programme outcome of these same participants would have been had they never participated This formula seems straightforward enough. However, directly obtaining data on the latter part of the equation – commonly referred to as the counterfactual – is logically impossible. This is because a person, household, community, etc. cannot simultaneously both participate and not participate in a programme. The counterfactual state of a programme’s participants can therefore never be observed directly; it can only be estimated. The randomised experiment is regarded by many as the most credible way of estimating the counterfactual, particularly when the number of units (e.g. people, households, or, in some cases, communities) that are being targeted is large. The random assignment of a sufficiently large number of such units to intervention and control groups should ensure that the statistical attributes of the two resulting groups are similar in terms of a) their pre-programmes outcomes (e.g. both groups have the same average incomes); and b) their observed characteristics (e.g. education levels) and unobserved characteristics (e.g. motivation) that affect the outcome variables of interest. In other words, randomisation works to ensure that the potential outcomes of both groups are the same. As a result – provided that threats such as differential attrition and intervention spill-over are minimal – any outcome differences observed at follow-up between the groups can be attributed to the workings of the programme. 10 Copperbelt Livelihoods Project Effectiveness Review Unfortunately – outside the context of specially designed pilot studies – randomised evaluation designs are seldom implemented in the context of social programmes, particularly in low-income countries. There can be cost, feasibility, and/or ethical constraints that militate against their use or simply the desire among implementing agencies to work with purposively chosen populations. Moreover, there are often cases where the opportunity to participate in a programme is put in place – as would be the case with the setting up of a micro-credit programme – and people choose whether to participate. Those that choose to participate are likely to be different to those that do not, including in characteristics that are intrinsically difficult to measure, e.g. motivation. 5.2 Alternative Evaluation Design Pursued There are several evaluation designs when the comparison group is non-equivalent that can identify reasonably precise intervention effect estimates. One solution is offered by matching: Find units in an external comparison group that possess the same characteristics, e.g. ethnicity, age, and sex, as those of the intervention group and match them on these characteristics. If matching is done properly in this way, the observed characteristics of the matched comparison group will be identical to those of the intervention group. The problem, however, with conventional matching methods is that with large numbers of characteristics on which to match, it is difficult to find comparators with similar combinations of characteristics for each of the units in the intervention group. The end result, typically, is that only a few units from the intervention and comparison groups get matched up, thereby, not only significantly reducing the size of the sample but also limiting the extent to which the findings can be generalised to all programme participants. Fortunately, matching on the basis of the propensity score – the conditional probability of being assigned to the programme group, given particular background variables or observed characteristics – offers a way out. The way propensity score matching (PSM) works is a follows: Units from both the intervention and comparison groups are pooled together. A statistical probability model is estimated, typically through logit or probit regression. This is used to estimate programme participation probabilities for all units in the pooled sample. Intervention and comparison units are then matched within certain ranges of their conditional probability scores. Tests are further carried out to assess whether the distributions of characteristics are similar in both groups after matching. If not, the matching bandwidth or calliper is repeatedly narrowed until the observed characteristics of the groups are statistically similar. Provided that a) the dataset in question is rich and of good quality; b) the groups possess many units with common characteristics (i.e. there is a large area of common support); and c) there are no unobserved differences lurking among the groups, particularly those associated with the outcomes of interest, PSM can produce good intervention effect estimates. Multivariable regression is another approach that is also used to control for measured differences between intervention and comparison groups. It operates differently from PSM in that it seeks to isolate the variation in the outcome variable explained by being in the intervention group net of other explanatory variables (key factors that explain variability in outcome) included the model. In this way, multivariable regression controls for measured differences between the intervention and comparison group. The validity of both PSM and multivariable regression are founded heavily on the “selection on observables” assumption, and therefore 11 Two popular methods were used to address selection bias – propensity score matching and multivariable regression. Copperbelt Livelihoods Project Effectiveness Review treatment effect estimates can be biased if there are unmeasured (or improperly measures) but relevant differences existing between the groups. Both PSM and multivariable regression were employed during data analysis, and efforts were made to capture key explanatory variables believed to be relevant in terms of the assessed outcomes, e.g. sex and age of household head (see Section 6.0 below). While no baseline data were available, efforts were made to reconstruct it through respondent recall. This method does have limitations, such as memory failure, confusion between time periods and so on. However, for data that can be sensibly recalled (e.g. ownership of particular household assets), it can serve to enhance the validity of a cross-sectional impact evaluation design. The reconstructed baseline data were used in two ways. First, several of the variables included in the PSM and regression procedures were baseline variables constructed from recalled baseline data. One set of variables, for example, was related to the respondent’s wealth status at baseline (e.g. whether they were asset rich, asset poor, or somewhere in between). This was done in attempt to control for baseline wealth differences between the intervention and comparison groups. The second way it was used was to derive pseudo difference-in-differences intervention effect estimates. With longitudinal or panel data, this is implemented by subtracting each unit’s baseline measure of outcome from its end line measure of outcome (end line outcome status minus baseline outcome status). The intention here is to control for time invariant differences between the groups. Bearing in mind the limitations associated with recalled baseline data, using PSM and/or regression and the difference-indifferences approaches together is considered to be a strong evaluation design. 5.3 The Comparison Group A key factor in ensuring the validity of any non-randomised impact evaluation design is to employ an appropriate comparison group. This is particularly true for ex-post, cross-sectional designs. Comparators that differ in relation to the baseline status of the outcome variable(s) of interest and/or who are subjected to different external events and influences will likely result in misleading conclusions about programme impact. Identifying a plausible comparison group is therefore critically important and is, generally speaking, not an easy task. As the project was delivered to all members residing in the 10 targeted communities, we were not confronted with the problem of people self-selecting themselves into the project. However, we did need to identify women who were similar to the intervention women but had not indirectly benefiting from the project as a result of spill over effects. Working together with local Oxfam staff and SAP partner staff, two comparison communities were identified which were deemed to meet these criteria. Unfortunately, due to time constraints and a lack of information on the individuals within these two communities, we were unable to randomly sample individual participants in advance of the survey. As an alternative, two days prior to the start of the survey, all women in the two communities were asked to be mobilised to a central point on the first day of surveying. From these groups, a random sample of women was chosen to be included in the survey. The comparison group comprised of women from two neighbouring communities. 12 Copperbelt Livelihoods Project Effectiveness Review 6. Methods of Data Collection and Analysis 6.1 Data Collection A household questionnaire was developed and revised locally to capture data on both the outcome variables presented in Section 4 above, as well as other key characteristics of the targeted and comparison women to implement the evaluation design described in Section 5. It was pre-tested by the local Oxfam team, partners and consultant, and subsequently revised. Potential enumerators were identified by the SAP team and 18 completed a two-day training course, which was led by a local Consulted but also supported by both the Oxfam/SAP staff. The second day involved a practice run at administering the question in a test community, where the performance of the enumerators was critically reviewed. This resulted in disengaging six of the enumerator trainees. Beneficiary listings for each of the 10 supported communities were used to carry out probability proportionate to size (PPS) sampling. The initial plan was to interview 160 women across the 10 Oxfam-supported communities. The sample size for the comparison groups was determined by increasing the sample size of the supported women by 50 percent – therefore a target of 240 women. Given that unmatched comparison data are given less weight or discarded altogether in PSM, it is preferable to have larger sample sizes for the comparison group. The number of women actually interviewed was 421 in total – 173 in the intervention group and 248 in the comparison group. A questionnaire was administered to a total of 421 women by 12 trained and locally recruited enumerators. 6.2 Data Analysis OGB developed data entry tools in Adobe Acrobat Pro, and the Consultant recruited and supervised data entry clerks to enter the data. After identifying and rectifying minor errors in MS Excel, the data were then imported into Stata for analysis, the results of which are presented in the following sections. Most of the analyses involved group mean comparisons using t-tests, as well as PSM with Stata’s psmatch2 module and various regression approaches. Kernel and nearest neighbour matching without replacement were the main methods used in implementing PSM. Variables used in the matching process were identified by first using backwards stepwise regression to identifying those variables that are correlated with the outcome measure of interest at a p-value of less than 0.20. The short-listed variables were then put into another stepwise regression model to identify those that are correlated with being a member of the intervention group. Covariate balance was checked following the implementation of each matching procedure. When covariate imbalance at p-values of 0.20 or less was identified, the bandwidth or calliper was reduced and the PSM procedure and covariate balance test implemented again. This was continued until all covariates were balanced at pvalues greater than 0.20. Boot-strapped standard errors enabled the generation of confidence intervals to assess the statistical significant of the effect sizes. The covariates as presented in Table 7.1 below were included in the various regression approaches undertaken, i.e. regression with robust standard errors, robust regression (to reduce the influence of outliers), and regression with control functions (to attempt to control for unobserved differences between the intervention and comparison groups). 13 Copperbelt Livelihoods Project Effectiveness Review 6.3 Main Problems and Constraints Encountered Overall, despite the usual hardships encountered when undertaking such intensive work, the data collection process went well. However, several challenges were encountered. These include: Significant differences observed between the intervention and comparison groups. As is presented below, the women interviewed in the comparison group are different, on average, across several of the variables. While these observable differences were controlled for during data analysis, it is likely that there are unobserved differences between the intervention and comparison groups as well. The effect estimates presented below, therefore, must be interpreted cautiously. Process for sampling the comparison group. This was mentioned in Section 5.3, but there may be sources of bias in mobilising women to a central location. For example, women living further away from the community centre, women who are less mobile, or have other constraints related to being away from the household, may not be as well represented. The comparison women were also only drawn from two villages. These villages may have happened to be better or worse off that the intervention villages for various reasons, e.g. differences in community leadership. Drawing data from a larger number of comparison areas would have helped to mitigate the bias that this may have introduced. 7. Results 7.1 General Characteristics Table 7.1 presents descriptive statistics for general household characteristics obtained through the administration of the questionnaire to the sampled women from both the intervention and comparison groups. The stars beside the number indicate differences between the two groups that are statistically significant at a 90 percent confidence level or greater. As is evident, there are several statistically significant differences between the groups, including those related to: Household size, including number of productive adults. Overall, the households of the intervention group are slightly larger in size. The number of adults, including those that are productive and unproductive, is also significantly greater. Age of household head. The average age of the household head is greater in the intervention group by approximately 6.5 years. Education levels. A household in the intervention group is more likely to have an adult with secondary education. Occupations of household members. Households of the intervention group are more likely to rear livestock, but are less likely to be involved in casual or waged labour. Distance from main district road. Members of the intervention group are more likely to be further from a main district road. Baseline wealth status. The intervention group is both more likely to be in the asset rich group at baseline and less likely to be in the asset poor group. The intervention and comparison women were found to be different in a number of important respects. 14 Copperbelt Livelihoods Project Effectiveness Review Table 7.1: Descriptive Statistics: Intervention and Comparison Women Interviewed Overall mean HH Size # of adults # of children # of young children # of productive adults # of unproductive adults # of dependents Female headed HH All elderly adults in HH Single adult HH Age of HH head HH head has sec educ HH adult with sec educ HH farms HH rears livestock HH runs IGA HH does casual labour HH does waged labour Asset poor baseline Asset middle baseline Asset rich baseline Far from main district road Low density rural population Observations * p<0.10, ** p<0.05, *** p<0.01 6.2399 2.8385 3.4014 2.1568 2.7197 0.1164 2.2732 0.3278 0.0546 0.1283 51.9454 0.3515 0.6152 0.9905 0.2257 0.3800 0.6698 0.1164 0.3349 0.3325 0.3325 0.7363 0.6295 421 Intervention mean 6.7052 3.1503 3.5607 2.1734 2.9711 0.1734 2.3468 0.3410 0.0578 0.0983 55.8439 0.3410 0.7225 1.0000 0.4277 0.3584 0.5780 0.0809 0.2659 0.3353 0.3988 0.9133 0.6590 173 Comparison mean 5.9153 2.6210 3.2903 2.1452 2.5444 0.0766 2.2218 0.3185 0.0524 0.1492 49.2258 0.3589 0.5403 0.9839 0.0847 0.3952 0.7339 0.1411 0.3831 0.3306 0.2863 0.6129 0.6089 248 Difference 0.7899*** 0.5293*** 0.2704 0.0282 0.4267*** 0.0968** 0.1250 0.0225 0.0054 -0.0509 6.6181*** -0.0178 0.1822*** 0.0161* 0.3431*** -0.0368 -0.1558*** -0.0602* -0.1172** 0.0046 0.1126** 0.3004*** 0.0501 t-statistic 3.17 3.66 1.30 0.17 3.08 2.23 0.73 0.48 0.24 -1.54 4.73 -0.38 3.84 1.68 9.03 -0.76 -3.38 -1.90 -2.52 0.10 2.42 7.29 1.05 7.2 Intervention Exposure The respondents were asked a number of questions about the extent they were exposed to the Oxfam supported interventions since 2008. Figure 7.1 presents the percentage of respondents that reported making use of the following types of support provided by external organisations: Agricultural inputs Agricultural training Maize storage facility Visit/support from agricultural extension worker Advocacy training Gender issues training Figure 7.1: % of households receiving support from external organisations 100% 80% 60% 40% 20% 0% Agricultural Agricultural Maize inputs training storage facility Support from ag worker Advocacy training Gender issues training The supported women reported receiving more support in all areas. Overall Intervention Comparison 15 Copperbelt Livelihoods Project Effectiveness Review As is apparent for all types of external support, there is a large difference between the households of the intervention and comparison groups. Women in households associated with the former, in particular, reported having been exposed to such interventions to a much greater extent. The statistics presented in Table 7.2 show that these differences are all highly statistically significant. Table 7.2: Differences in support received from external organisations since 2008 Overall mean Agricultural inputs Agricultural training Maize storage facility Support from ag. worker Advocacy training Gender issues training Observations * p<0.1, ** p<0.05, *** p<0.01 0.45 0.48 0.23 0.40 0.32 0.37 421 Intervention mean 0.97 0.98 0.54 0.88 0.74 0.81 173 Comparison mean 0.09 0.13 0.02 0.06 0.03 0.06 248 Difference 0.8726*** 0.8478*** 0.5174*** 0.8237*** 0.7076*** 0.7528*** t-statistic 34.92 31.08 15.45 30.13 22.83 24.62 Table 7.3 presents the differences in either the number of times a household has received a certain type of support and the number of household members trained in a particular activity. As is apparent from the table, women in households in the intervention group benefited to a greater extent in all areas of inputs received or household members trained, when compared with the comparison group. All of these differences are highly statistically significant. Table 7.3: Differences in number of times support received from external organisations, or household members trained since 2008 Overall mean # of times receiving agricultural inputs # of HH members trained in agric. # of maize storage facilities received # of visits from agric. Worker # of HH members trained in advocacy # of HH members trained in gender issues Observations * p<0.1, ** p<0.05, *** p<0.01 1.9097 1.0451 0.9976 2.5024 0.7150 0.8195 421 Intervention mean 3.8844 1.7514 1.7110 5.2500 1.2717 1.4046 173 Comparison mean 0.5323 0.5524 0.5000 0.5968 0.3266 0.4113 248 Difference 3.3521*** 1.1990*** 1.2110*** 4.6532*** 0.9451*** 0.9933*** t-statistic 11.93 8.83 7.41 10.28 10.46 7.76 7.3 Global Outcome Indicator for Women’s Empowerment Recall OGB’s global outcome indicator for women’s empowerment: % of supported women meaningfully involved in household decision making and influencing affairs at the community level. 16 Copperbelt Livelihoods Project Effectiveness Review As discussed above, this indicator is comprised of two dimensions – women’s involvement in household decision-making and their perceived ability to influence how the affairs of their communities are governed. The former was measuring by asking the respondents about the degree of their involvement in 25 household decision-making areas. Responses were scored from zero (no involvement) to 4 (sole decision-maker), and a percentage score was calculated for the respondent’s overall involvement. To measure the second dimension, the women were asked the extent to which they agree or disagree with various statements about their ability to influence affairs in their communities (see above). For a woman to score positively on the global indicator, she had to score well on both indicators. In particular, she was coded with 1 if she obtained a score of 67% or higher for the household decision-making measure and a score of 60% or higher for community influencing measure. Table 7.4 presents the result of a comparison between the intervention and comparison women. As is evident, only one of the adjusted effect estimates is statistically significant. There is therefore little evidence that the project successfully affected the global indicator. Table 7.4: % of women meaningfully involved in household decision making and influencing affairs at the community level Overall Unadjusted Sample mean Intervention mean Comparison mean Unadjusted difference Observations PSM (ATT) Post matching difference (kernel) Observations Post matching difference (no replacement) Observations Multivariable regression MVR coefficient (probit): Observations MVR coefficient (probit) with control functions: Observations t statistics in parentheses * p<0.10, ** p<0.05, *** p<0.01 PSM estimates bootstrapped 1000 repetitions 0.37 0.41 0.35 0.0608 (1.26) 419 0.0731 (1.50) 419 0.0983** (2.06) 419 0.1040 (1.56) 419 0.1113 (1.58) 415 After controlling for differences between the groups, only one of the effect estimates is statistically significant. 17 Copperbelt Livelihoods Project Effectiveness Review 7.4 Household decision-making The impact of the project on women’s involvement in household decision-making can also be examined in isolation. Table 7.5 present the percentage of women who scored positively in relation to household decision-making (i.e. those having a score of 67% or higher). As is apparent, the proportions in the intervention and comparison group are very similar at 61% and 60% respectively, providing no evidence that the project has positive affected this dimension. Table 7.5 confirms that this small difference is not statistically significant at a 90% level for both unadjusted and adjusted figures. Table 7.5: % of women meaningfully involved in household decision making No difference was found between the intervention and comparison women for the household decision-making measure. Overall Unadjusted Sample mean Intervention mean Comparison mean Unadjusted difference Observations PSM (ATT) Post matching difference (kernel) Observations Post matching difference (no replacement) Observations Multivariable regression MVR coefficient (probit): Observations MVR coefficient (probit) with control functions: Observations 0.60 0.61 0.60 0.0263 (0.21) 421 0.0239 (0.42) 421 0.0427 (0.76) 412 0.0019 (0.03) 421 0.0157 (0.22) 417 t statistics in parentheses * p<0.10, ** p<0.05, *** p<0.01 PSM estimates bootstrapped 1000 repetitions One may also wonder if the results would be different if responses from femaleheaded households were removed from the analysis, given that women of such households will be more likely to wield greater decision-making power. The results are presented in Table 7.6, and, again, provide no evidence of impact. 18 Copperbelt Livelihoods Project Effectiveness Review Table 7.6: % of women meaningfully involved in household decision making (non femaleheaded households) Overall Unadjusted Sample mean Intervention mean Comparison mean Unadjusted difference Observations PSM (ATT) Post matching difference (kernel) Observations Post matching difference (no replacement) Observations Multivariable regression MVR coefficient (probit): Observations MVR coefficient (probit) with control functions: Observations t statistics in parentheses * p<0.10, ** p<0.05, *** p<0.01 The results don’t change even when male-headed households are examined in isolation. 0.43 0.44 0.43 0.0319 (0.21) 283 0.0538 (0.78) 283 0.0467 (0.69) 276 0.0073 (0.09) 282 -0.0095 (-0.11) 280 PSM estimates bootstrapped 1000 repetitions 7.5 Perceived ability to influence affairs at the community level As is the case for household decision-making, it is also possible to examine women’s perceived ability to influence affairs in their communities in isolation. Figure 7.2 presents the percentages of women who scored positively for this dimension of the global indicator (i.e. those having a score of 60% or higher). Figure 7.2: % of women influencing affairs at the community level (unadjusted) 80% 70% 60% 50% 40% 30% 20% 10% 0% Overall Intervention Comparison 19 Copperbelt Livelihoods Project Effectiveness Review As is evident, the proportion of women who perceive they are able to influence affairs at the community level is higher for the intervention group (69%) as compared with the comparison group (53%). Table 7.7 reveals that these differences are statistically significant across all of the adjustment procedures. Table 7.7: % of women influencing affairs at the community level Overall Unadjusted Sample mean Intervention mean Comparison mean Unadjusted difference 0.60 0.69 0.53 0.165*** (3.43) 419 0.127** (2.55) 416 0.157*** (2.95) 412 0.186*** (2.75) 416 0.176*** (2.59) 412 Women in the intervention villages are more likely to report being in a position to influence how their communities are governed. Observations PSM (ATT) Post matching difference (kernel) Observations Post matching difference (no replacement) Observations Multivariable regression MVR coefficient (probit): Observations MVR coefficient (probit) with control functions: Observations t statistics in parentheses * p<0.10, ** p<0.05, *** p<0.01 PSM estimates bootstrapped 1000 repetitions Table 7.8 presents these findings based on the percentage scores of each respondent for this indicator. The table show that the average score for women in the intervention group is 67% compared to 60% in the comparison group. It is clear that the differences between the intervention and comparison women remain highly significant across all the statistical adjustment procedures. 20 Copperbelt Livelihoods Project Effectiveness Review Table 7.8: Score for influencing affairs at the community level Overall Unadjusted Sample mean Intervention mean Comparison mean Unadjusted difference Observations PSM (ATT) Post matching difference (kernel) Observations Post matching difference (no replacement) Observations Multivariable regression MVR coefficient (robust): Observations MVR coefficient (rreg): Observations MVR coefficient with control functions: Observations t statistics in parentheses * p<0.10, ** p<0.05, *** p<0.01 PSM estimates bootstrapped 1000 repetitions 0.63 0.67 0.60 0.0630*** (6.62) 419 0.0635*** (5.21) 352 0.0647*** (5.06) 346 0.0575*** (4.58) 352 0.0505*** (4.10) 352 0.0583*** (4.58) 348 7.6 Women’s self-efficacy The respondents were also asked how they felt they could cope with particular situations as a measure of their self-efficacy. The women, in particular, were read eight statements and asked whether the statement was ‘not at all true’, ‘hardly true’, ‘moderately true’ or ‘exactly true’. A score was then generated from their responses, with a maximum score of 32. The results of a comparison of the intervention and comparison women in relation to this score are presented in Table 7.9. The average score for the women in the intervention group was 23.5 compared to 25.8 in the comparison group. After controlling for differences between the groups, four out of the five adjusted effect estimates indicate that the comparison women have higher self efficacy compared with the intervention women. Factor scores were also constructed for the women’s responses to the various Likert scales, and the intervention and comparison women were again compared. Unfortunately, the results present a similar picture. 21 Copperbelt Livelihoods Project Effectiveness Review Table 7.9: Women’s self-efficacy scores Overall Unadjusted Sample mean Intervention mean Comparison mean Unadjusted difference 24.8 23.5 25.8 -2.229*** (-5.17) 408 -0.883 (-1.22) 366 -1.416** (-2.36) 360 -1.240** (-2.03) 366 -2.057*** (-3.83) 365 -1.311** (-2.05) 362 The comparison women were found to have higher selfefficacy scores than the intervention women. Observations PSM (ATT) Post matching difference (kernel) Observations Post matching difference (no replacement) Observations Multivariable regression MVR coefficient (robust): Observations MVR coefficient (rreg): Observations MVR coefficient with control functions: Observations t statistics in parentheses * p<0.10, ** p<0.05, *** p<0.01 PSM estimates bootstrapped 1000 repetitions 7.7 Female asset ownership As discussed in Section 3, it is assumed that increasing women’s access to, and control over, productive assets is an important way of promoting their empowerment. As part of the questionnaire, the female respondents were asked who owns the following assets: Land Livestock Home Other house Major farming assets Major off-farm income generating assets (e.g. sewing machine) Savings Any other valuable assets (e.g. TV, jewellery, furniture) For each of these assets the respondents were asked whether they owned fully, owned jointly, or did not own. Figure 7.3 illustrates that 66% of women in the intervention group owned at least one strategic asset, compared to 50% in the comparison group. 22 Copperbelt Livelihoods Project Effectiveness Review Figure 7.3: % of women owning at least one strategic asset (unadjusted) 70% 60% 50% 40% 30% 20% 10% 0% Overall Intervention Comparison Table 7.10 reveals that this difference is statistically significant, even after controlling for observable differences between the intervention and comparison women. Table 7.10: % of women owning at least one strategic asset Overall Unadjusted Sample mean Intervention mean Comparison mean Unadjusted difference Observations PSM (ATT) Post matching difference (kernel) Observations Post matching difference (no replacement) Observations Multivariable regression MVR coefficient (robust): Observations MVR coefficient with control functions: Observations t statistics in parentheses * p<0.10, ** p<0.05, *** p<0.01 PSM estimates bootstrapped 1000 repetitions The proportion of women owning at least one strategic asset is higher in the intervention group. 0.57 0.66 0.50 0.159*** (3.24) 421 0.159** (1.98) 359 0.140** (2.08) 348 0.191** (2.55) 359 0.185** (2.44) 355 Table 7.11 presents the same analysis, but this time excludes female headed households. We can see that the statistically significant differences between the intervention and comparison women still hold. 23 Copperbelt Livelihoods Project Effectiveness Review Table 7.11: % of women owning at least one strategic asset (excluding female headed households) Overall Unadjusted Sample mean Intervention mean Comparison mean Unadjusted difference Observations PSM (ATT) Post matching difference (kernel) Observations Post matching difference (no replacement) Observations Multivariable regression MVR coefficient (robust): Observations MVR coefficient with control functions: Observations t statistics in parentheses * p<0.10, ** p<0.05, *** p<0.01 PSM estimates bootstrapped 1000 repetitions 0.39 0.50 0.31 0.192*** (3.31) 283 0.172*** (2.58) 258 0.157** (2.23) 258 0.180** (2.02) 255 0.195** (2.12) 255 7.8 Household asset ownership As outlined in Section 3, one of the key outcomes for this project is an improvement in household income. While the women’s empowerment questionnaire used in this exercise does not include explicit measures of household income, we are able to analyse the change in household assets as a proxy measure for changes in household wealth status. Households that are wealthy tend to have more tangible material possessions or other locally relevant wealth indicators, such as livestock, tin roofs (as opposed to grass), bicycles, radios, cemented floors (as opposed to dirt), etc. Efforts were therefore made to capture data on household wealth indicators, particularly in relation to those assumed to be relevant to differentiating the better and worse off in the intervention and comparison communities. Respondents were asked to report on the various wealth indicators at both the time of review, as well as for the baseline period (2008), thereby attempting to reconstruct baseline data. The specific household wealth indicators are presented in Table 7.12. Where sensible, efforts were made to capture not only on whether the household had the asset in question, but also the specific number owned. In addition, for indicators such as those related to material used to construct specific features of the respondents’ homes scores were allocated depending on the material in question. For example, 0 points was given for respondents reporting the floor of their homes were made of dirt, 1 point if it was made from cement or unfinished wood, and 2 points for tiles, vinyl, or finished wood. Respondents were asked to report on ownership of various household assets, both for the present time and baseline (2008). 24 Copperbelt Livelihoods Project Effectiveness Review Table 7.12: List of assets and other wealth indicators used in the measurement of HH asset ownership Electricity (inc. solar/generator) Lamps – electric, kerosene or other Television Clock/watch Table Radio DVD Iron Toilet type Material of roof of home Bed Phone Bicycle Motorcycle Wheel barrow Vehicle Hoe Axe Material of floors of home Area of agricultural land Sewing machine Milling machine Plough Cart Cattle Goat Pig Poultry Material of walls of home Number of rooms in household The numbers of assets owned were then grouped into three quantiles to avoid the analysis being overly influenced by extreme values. Principal Component Analysis (PCA) was then run on all the wealth indicators presented in Table 7.14, and an asset index was created based on the first principal component that was generated. This was done for both sets of indicators associated with the endline and baseline periods. Table 7.1 indicated significant differences between the intervention and comparison group in terms of whether respondents belonged to the ‘asset poor’ or ‘asset rich’ group at baseline. However, using matching and the ‘difference in difference’ method, we are able to examine the changes in wealth indicators between 2008 and 2011 for both groups to test whether there are differences between the groups in relation to these change. Once the changes in wealth indicators were calculated, the differences were grouped into three quantiles and PCA was run on these ‘differenced quantiles’. The results are presented in Table 7.13. Table 7.13: Changes in asset ownership Overall Unadjusted Sample mean Intervention mean Comparison mean Unadjusted difference Observations PSM (ATT) Post matching difference (kernel) Observations Post matching difference (no replacement) Observations Multivariable regression MVR coefficient (fe): Observations MVR coefficient (fe) with control functions: Observations t statistics in parentheses * p<0.10, ** p<0.05, *** p<0.01 PSM estimates bootstrapped 1000 repetitions Households in the intervention group gained more in asset wealth over the lifespan of the project. 0.00 0.50 -0.35 0.849*** (4.67) 420 0.763*** (3.00) 382 0.459** (2.05) 359 0.681*** (3.39) 378 0.711*** (3.46) 382 25 Copperbelt Livelihoods Project Effectiveness Review As is evident, the intervention households gained more in asset wealth as compared with the comparison group. 7.9 Production of maize One of the specific interventions of the project (see Section 3) was to provide maize seed and fertiliser, together with training and agricultural extension support. As part of the questionnaire, all respondents were asked to estimate the kilograms of maize produced by the household over the last 12 months compared with 2008 (baseline). Using this information we are able to assess the impact of the project on maize production. More significant increases in maize production were observed for the intervention households. Figure 7.4: Average kilograms of maize produced per household 1400 1200 1000 800 Maize (kg) 2008 600 Maize (kg) now 400 200 0 Intervention Comparison Figure 7.4 illustrates that maize production in the intervention households increased on average from 809kg in 2008 to 1,218 kg in 2011. The equivalent figures for the comparison group are 680kg in 2008 to 814kg in 2011. The results in Table 7.14 show that these differences are statistically significant for 4 of the 5 estimation methods. However, the robust regression method, which reduces the influence of outliers, suggests there is no significant difference for the majority of households. As a result, there is no significant evidence that the project has positively impacted maize production for the typical intervention household. The results are the same when maize growers are analysed in isolation (see Table 7.15). 26 Copperbelt Livelihoods Project Effectiveness Review Table 7.14: Difference in household maize production (kgs) between 2008 and 2011 Overall Unadjusted Sample mean Intervention mean Comparison mean Unadjusted difference Observations PSM (ATT) Post matching difference (kernel) Observations Post matching difference (no replacement) Observations Multivariable regression MVR coefficient (fe): Observations MVR coefficient (rreg): Observations MVR coefficient (fe) with control functions: Observations t statistics in parentheses * p<0.10, ** p<0.05, *** p<0.01 The apparent impact of the project on maize production is due to the influence of outliers. 245.22 409.36 130.72 278.6*** (3.23) 421 315.9*** (2.86) 373 292.1** (2.48) 361 242.5** (2.04) 373 38.2 (0.61) 373 246.2** (2.07) 369 PSM estimates bootstrapped 1000 repetitions Table 7.15: Difference in household maize production (kgs) between 2008 and 2011 (maize growers only) Overall Unadjusted Sample mean Intervention mean Comparison mean Unadjusted difference Observations PSM (ATT) Post matching difference (kernel) Observations Post matching difference (no replacement) Observations Multivariable regression MVR coefficient (robust, fe): Observations MVR coefficient (rreg): Observations MVR coefficient (fe) with control functions: Observations t statistics in parentheses * p<0.10, ** p<0.05, *** p<0.01 293.29 424.48 171.52 263.0*** (2.69) 352 481.0*** (3.48) 316 314.4** (2.44) 287 280.6** (2.17) 316 118.2 (1.40) 316 290.7** (2.24) 315 PSM estimates bootstrapped 1000 repetitions 27 Copperbelt Livelihoods Project Effectiveness Review 7.10 Production of groundnuts Together with maize seed and fertiliser, ground nut seeds were also provided to households in the intervention group. As with maize, all respondents were asked to estimate the kilograms of groundnuts produced by the household over the last 12 months compared with 2008 (baseline). Using this information we are able to analyse the impact of the project on groundnut production in the intervention group compared with the comparison group. Figure 7.5: Average kilograms of groundnuts produced by households 300 250 200 150 100 50 0 Intervention Comparison Groundnuts (kg) 2008 Groundnuts (kg) now Figure 7.5 illustrates that groundnut production in the intervention group households increased on average from 151kg in 2008 to 248 kg in the year leading up to the review. The equivalent figures for the comparison group are 64kg in 2008 to 68kg in the year leading up to the review. As the results in Table 7.16 show, there is evidence to show that these differences are significant. Table 7.16: Difference in household groundnut production (kgs) between 2008 and 20011 Overall Unadjusted Sample mean Intervention mean Comparison mean Unadjusted difference Observations PSM (ATT) Post matching difference (kernel) Observations Post matching difference (no replacement) Observations Multivariable regression MVR coefficient (fe): Observations MVR coefficient (fe) with control functions: Observations t statistics in parentheses * p<0.10, ** p<0.05, *** p<0.01 There is modest evidence that the project has increased groundnut production. 42.13 97.04 3.82 93.22* (1.84) 421 88.50* (1.71) 419 89.24* (1.67) 419 67.65* (1.70) 419 49.74 (1.41) 415 PSM estimates bootstrapped 1000 repetitions 28 Copperbelt Livelihoods Project Effectiveness Review Table 7.17: Difference in household groundnut production (kgs) between 2008 and 2011 – growers only Overall Unadjusted Sample mean Intervention mean Comparison mean Unadjusted difference Observations PSM (ATT) Post matching difference (kernel) Observations Post matching difference (no replacement) Observations Multivariable regression MVR coefficient (fe): Observations MVR coefficient (rreg): Observations MVR coefficient (fe) with control functions: Observations t statistics in parentheses * p<0.10, ** p<0.05, *** p<0.01 There is no statistically significant difference between the intervention and comparison households when those growing groundnuts at baseline are examined separately. 86.10 130.14 12.31 117.8 (1.55) 206 111.8 (1.32) 197 3.315 (0.05) 142 90.13 (1.40) 197 13.24 (0.36) 197 65.17 (1.04) 197 PSM estimates bootstrapped 1000 repetitions 7.11 Income earned from selling vegetables Another key intervention of the project was the provision of vegetable seed and fertiliser for winter crop production (see Section 3). Training was also provided by project staff and followed up extension support provided. As part of the questionnaire, respondents were asked to estimate the income earned from selling vegetables over the 12 months, as compared with 2008 (baseline). Using this information we are able to analyse the impact of the project on income raised from selling vegetables in the intervention group compared with the comparison group. Figure 7.6: Average income earned from selling vegetables by households 450,000 400,000 350,000 300,000 250,000 200,000 150,000 100,000 50,000 0 Intervention Comparison Income from vegetables (ZMK), 2008 Income from vegetables (ZMK), now 29 Copperbelt Livelihoods Project Effectiveness Review Figure 7.6 illustrates that income earned from selling vegetables in the intervention group households increased on average from 274,000 Zambian Kwacha (ZMK) in 2008 to 396,000 ZMK in the year leading up to the review. The equivalent figures for the comparison group are 64,000 ZMK in 2008 to 139,000 ZMK in the year leading up to the review. As the results in Table 7.18 show, there is no evidence to show that these differences are significant. Table 7.18: Difference in income earned from selling vegetables between 2008 and the year leading up to the review Overall There is no evidence that the project bolstered income earned through the selling of vegetables. Unadjusted Sample mean Intervention mean Comparison mean Unadjusted difference Observations PSM (ATT) Post matching difference (kernel) Observations Post matching difference (no replacement) Observations Multivariable regression MVR coefficient (fe): Observations MVR coefficient (fe) with control functions: Observations t statistics in parentheses * p<0.10, ** p<0.05, *** p<0.01 PSM estimates bootstrapped 1000 repetitions 94,414.25 121,828.90 75,290.32 46,538.6 (0.95) 421 -7,209.8 (-0.13) 382 -37,270.3 (-0.55) 376 -4,370.6 (-0.10) 382 4,125.3 (0.11) 378 8. Conclusion and Programme Learning Considerations 8.1 Conclusions Overall, the effectiveness review found a mixed picture in terms of differences between the interviewed women and their comparators in relation to the following outcomes: a) Women’s involvement in household decision-making and their perceived ability to influence the governing of community affairs b) Women’s self efficacy c) Increased ownership of strategic assets by women d) Improved household income (as measured by household asset ownership) e) Improved agricultural production 30 Copperbelt Livelihoods Project Effectiveness Review Overall, there is no evidence that the project significantly affected OGB’s global indicator for women’s empowerment. However, when this indicator is broken down by dimension, a different picture emerges. There is evidence that the project performed well in some areas but not so well in others. When women’s involvement in household decision making is examined in isolation, for instance, there is very little difference between the groups. The proportion of women meaningfully involved in household decision making in the intervention group is 61%, compared to 60% in the comparison group. However, significant differences were identified in women’s perceptions of how they can influence decisions in the community. Sixty-nine percent of women in the intervention group felt able to influence community level decisions and affairs, compared to 53% of women in the comparison group – a statistically significant difference. For the self-efficacy measure, the comparison women were actually found to be better off than the intervention women. Given the non-experimental nature of the data, we cannot confidently conclude the project had a negative impact on women’s self-efficacy. However, there is certainly no evidence to suggest that it had a positive impact in this regard. The project appears to have performed better in relation to women’s ownership of strategic assets. Overall, two-thirds of women in the intervention group owned at least one strategic asset, compared to 50% in the comparison group – a difference which remained significant after controlling for observed differences between the groups. Moreover, there is evidence that the project successfully increased household asset wealth. In particular, households in the intervention group experienced greater gains in asset ownership over the life span of the project. It is difficult to explain, however, how this change may have been brought about through the project. As per the project’s theory of change, changes in household income and wealth status were to come about through increasing agricultural projection. However, there is not significant evidence that it was successful in this regard, particularly for the majority of intervention households. On explanation may be due to the fact that the project distributed tools. However, further interrogation of the data would be required to either confirm or rule out this possibility. 8.2 Programme Learning Considerations The findings and learning considerations in this report are based on the quantitative research exercise. It would be beneficial to have a deeper qualitative understanding of the context and causal factors underlying the review’s findings. We therefore propose a collaborative process between Oxfam advisers and the programme team to discuss the findings and learning considerations in order to forge a way forward to benefit both this project and future work of this type. Initial learning considerations emerging from the analyses of the data include: Review options for making women’s empowerment interventions more explicit and tangible in project design. The project was selected for review primarily to understand the extent it has contributed to empowering the supported women. However, its primary objective was to improve the livelihoods of the targeted farmers, 60% of whom were women. There is evidence to suggest that it was successful in 31 Copperbelt Livelihoods Project Effectiveness Review More developed strategies to empower women should be built into projects of this nature. this regard. This finding would have been more conclusive if data were collected on other measures relating to household income, e.g. household expenditure and food security, as well as a plausible causal explanation for how the change came about, e.g. evidence that the project was successful in bolstering agricultural production. Despite that women’s empowerment was not the primary focus of the project, there is evidence that it positively affected two measures of women’s empowerment – women’s ability to influence community governance and ownership of strategic assets. At the same, time it did not perform well for the two other measures – involvement in household decision making and self-efficacy. Given this, the project would benefit from a more strategic approach to empowering women. The women’s economic leadership team in Oxfam have identified four key factors that are believed to influence women’s economic empowerment: Women’s asset ownership Women’s increased income Women’s skills and knowledge Attitudes/beliefs – women’s and men’s perception of women’s economic contribution We recommend engaging Oxfam’s women’s economic empowerment/ gender advisers, both in the region and at headquarters, to explore options for including or strengthening specific interventions which target each of these factors in current and future programming of this type. Consider increasing efforts to further bolster agricultural production and support the marketing of agricultural commodities. There is no evidence that the project significantly bolstered agricultural production or income for the majority of the targeted farmers. This is an area for further qualitative research to explore differences in household production and perhaps improve the targeting of resources and training accordingly. It is also recommended that a more strategic approach be considered to increase production and add value to the produce, promote collective action, and improve the marketing of crop products. This should be informed by an agri-business feasibility study which examines the comparative production advantage of the supported women and their families and market demand for the identified crops. One of the key issues may be in linking the agricultural produce with markets, especially as the supported communities are far from the main district centre and road. Discussion with Oxfam’s economic advisers may assist in highlighting specific market linkage interventions which would benefit this project. This may also support the empowerment of women as well. The livelihoods advisory team in Oxfam, for instance, has found that programmes are most successful in achieving women’s economic leadership when there is the ‘driver’ of a market opportunity that helps convince producer, organisations, buyers and other actors to work to overcome barriers that women face. It is therefore recommended that any market feasibility study fully incorporates a gendered market selection and mapping exercise in order to strengthen the opportunities for women’s economic empowerment. 32 Copperbelt Livelihoods Project Effectiveness Review Explore options to increase the effectiveness and impact of the project further by accompanying direct implementation with an advocacy strategy to make relevant policy and institutional changes. Making strides forward in improving women’s empowerment at a household and community level, while influencing policy at a national or sub-national level has the potential to drive more rapid change in how the role of women is viewed and enshrined in policy/legal frameworks. There is also scope to affect the enabling environment around economic development which could assist in driving greater economic growth for women producers. Consideration and analysis of a range of institutional factors appropriate to the local context is recommended, and Oxfam’s advocacy advisers may be able to assist in focusing effective action relating to: Land and property rights Gender roles and behaviours Social norms and informal networks Governance, commercial law and enforcement Trade rules and competition policy Economic infrastructure Natural environment and resources Consumer trends Quality standards and regulations Follow up on some of the specific findings from this report with further qualitative research. Focused qualitative research may help to explain why the supported women appear to be more empowered to influence affairs at a community level, but seemingly less empowered at a household level. It would also be interesting to investigate why the significant increase in asset ownership between 2008 and 2011 in the intervention group is not mirrored by large changes in agricultural production. 33