How are effectiveness reviews carried out?

Please download to get full document.

View again

of 6
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.
Information Report



Views: 16 | Pages: 6

Extension: PDF | Download: 0

Related documents
Since 2011, Oxfam Great Britain (GB) has been conducting rigorous evaluations annually on samples of randomly selected mature projects (projects which are a substantial way through implementation or have finished). The purpose of these evaluations is to help the organization understand and evidence whether its work is resulting in positive change in the lives of the women and men with whom and for whom it works. The Effectiveness Reviews consider Oxfam GB
  EFFECTIVENESS REVIEW SERIES ‘enhancing effectiveness through evidence-based learning’  Every year since 2011, Oxfam Great Britain (GB) has conducted rigorous evaluations on samples of randomly selected mature projects (projects that are a substantial way through implementation or have nished) to help the organisation understand and evidence whether its work is resulting in positive change in the lives of the women and men with whom and for whom it works.These ‘Effectiveness Reviews’ consider Oxfam GB’s work across six thematic areas:ã Humanitarian Responseã Accountabilityã Livelihoods supportã Resilience ã Women’s Empowermentã Good Governance (which brings together “Citizen voice” and “Policy Inuence”)Evaluations under the rst two thematic areas - humanitarian and accountability - consider the degree to which interventions meet agreed standards. Accountability Reviews consider the extent to which projects have met Oxfam’s standards for accountability to partners and communities. Evaluations of humanitarian responses consider the extent to which the response under review met internationally agreed humanitarian standards. Oxfam’s Effectiveness Review Series: Evaluations of the nal four thematic areas – Livelihoods, Resilience, Women’s Empowerment and Good Governance – aim to assess the impact of our projects. They consider whether there is evidence of a cause-effect relationship between the intervention and observed outcomes/ impact, even where the intervention is only one factor contributing to that change. Oxfam GB uses four distinct methodologies to evaluate projects selected as part of the Effectiveness Review series, as no one evaluation approach is appropriate for all programmes and contexts.   HOW ARE EFFECTIVENESS REVIEWS CARRIED OUT?  An enumerator in the Philippines carrying out an Effectiveness Review 2014/15 interview using a mobile phone. Photo: Emily Tomkys/OxfamTop: Climate change adaptation and advocacy project, Nepal, evaluated under resilience in 2012/13. Photo: Jisu Mok/Oxfam  More information can be found at Oxfam’s Policy and Practice website: For more information, contact Oxfam’s Programme Quality Team - WHAT WE ARE MEASURING Oxfam denes accountability as the process through which an organisation balances the needs of stakeholders in its decision-making and activities, and delivers against this commitment. This reects Oxfam’s belief that how   work is done is as important as what is done.  Accountability Reviews consider the extent to which projects are meeting their accountability commitments to partners and communities. They assess sampled projects that are approximately a quarter to half way through implementation – enough time for accountable relationships to have been developed and with enough time remaining to put learning into practice during the lifetime of the project.The assignments consider three key dimensions of accountability:ã Transparency – being transparent in our relationships.ã Feedback/listening – that feedback mechanisms exist and lead to change.ã Participation – key stakeholders participate in decision-making and implementation throughout the lifetime of a project.They also examine the leadership, systems and practices of Oxfam and partner staff, as well as the perceptions of staff, partners and communities. Questions about partnership practices, staff attitudes, and satisfaction (how useful the project is to the people and how wisely the money on the project has been spent) are asked where appropriate. EVALUATION DESIGN  A series of set exercises are carried out by external evaluators in order to collect qualitative and quantitative data from country and project teams, and partners and communities. Accountability Reviews usually consist of a document review, workshops, on-site focus group discussions with beneciary communities and structured interviews with Oxfam staff, partners and other stakeholders. The evaluator triangulates this information to reach conclusions on the degree to which Oxfam’s work meets its own standards for accountability.   ACCOUNTABILITY WHAT WE ARE MEASURING Evaluations conducted of our Humanitarian Responses consider the extent to which the intervention has met internationally agreed humanitarian standards. These are based on the hypothesis that a good-quality programme will lead to greater impact. EVALUATION DESIGN Large-scale humanitarian responses are evaluated using the Humanitarian Indicator Tool (HIT), a methodology designed to estimate the degree to which the programme meets the dimensions of the sector-wide and internationally recognised Core Humanitarian Standard. The HIT has 15 quality standards, each with dened benchmarks, which allow external evaluators to assess and score whether the standard was ‘met’, ”almost met”, ‘partially met’ or ‘not met’. Standards include issues such as timeliness; coverage of affected populations; technical aspects measured against Sphere standards; partner relationships; and the needs of women, girls, men, boys and vulnerable groups, and are weighted depending on their relative importance. The HIT is carried out as a desk study by an external evaluator using documented evidence to generate a score against each standard and a cumulative total expressed as a percentage. If there are gaps in the documentation, or clarity is needed, supplementary interviews are carried out by phone or Skype. HUMANITARIAN RESPONSE Collecting clean water during the response in Democratic Republic of Congo, evaluated in 2013/14. Photo: Kate Holt/OxfamYouth agents of change in Georgia  – My Rights, My Voice programme.  Accountability Review 2013/14. Photo: Khatuna Khubitia/Oxfam  Full version of this report and more information can be found at Oxfam’s Policy and Practice website: For more information, contact Oxfam’s Programme Quality Team -   More information can be found at Oxfam’s Policy and Practice website: For more information, contact Oxfam’s Programme Quality Team - WHAT WE ARE MEASURING LIVELIHOODS SUPPORT  Projects are evaluated for their impact on overall household income. Asking directly for all the details of a household’s income (which includes income from many different sources) and agricultural production (which may be consumed in the household rather than sold for cash; bartering, etc.) would be complicated and prone to error. Instead, measurement is made on the basis of household consumption and expenditure, which are normally found to be closely linked to income in low- and medium-income countries. Survey respondents are asked for details of the food consumed in the household during the seven days prior to the survey, as well as for expenditure on non-food items made by the household in the last month and 12 months. These data are aggregated to provide an indication of the household’s overall consumption per person, per day. Respondents are also asked about their ownership of various types of household goods and assets, and the condition of their housing. These data are then used to create a wealth index. Effectiveness Reviews that focus on livelihoods support also include measures of the projects’ impacts on intermediate outcomes. Depending on the nature of the project this may, for example, involve looking at the extent to which it has resulted in changes in the adoption of improved agricultural practices, the level of agricultural production or sales, the protability of household businesses, or households’ saving behaviour or use of credit. Examining these intermediate outcomes enables the evaluation to identify which aspects of the project’s theory of change have been successful, and where further intervention may be needed. RESILIENCE  These Effectiveness Reviews seek to understand the extent to which Oxfam projects have enabled people to minimise risk from shocks (whether natural disasters, conict, or more local-level or household- level crises) and adapt to emerging trends and uncertainty (such as changing climate conditions). A full evaluation of the outcomes of increased resilience can be conducted only after shocks, stresses and uncertainty have been experienced, where people’s and communities’ actual responses can be assessed. In practice, Effectiveness Reviews are often carried out in the absence of shocks, meaning that such an assessment cannot be made. Instead, the evaluation team seeks to measure context-specic indicators for characteristics that are thought to be associated with resilience in the context in question. A framework of ve dimensions is used to categorise these characteristics: ã Livelihood viability  – the extent to which a household’s livelihood can continue and thrive in spite of shocks, stresses, and uncertainty. ã Innovation potential  – the ability to take appropriate risks and positively adjust to change. ã Access to contingency resources and support  – access to back-up resources and appropriate assistance in times of crisis. ã Integrity of the natural and built environment  – the health of local ecosystems, natural resource management practices, and presence of appropriate physical infrastructure. ã Social and institutional capability  – the extent to which formal and informal institutions are able to ensure equitable access to essential information, services, and resources to reduce risk and support positive adaptation.   COMMUNITY DEVELOPMENT PROJECTS Drought Management Initiative: Livestock, Kenya. Evaluated under resilience in 2012/13. Photo: Alejandro Chaskielberg/OxfamIncreasing smallholder producers’ access to markets in Colombia, evaluated under livelihoods in 2013/14. Photo: Rob Fuller/Oxfam  More information can be found at Oxfam’s Policy and Practice website: For more information, contact Oxfam’s Programme Quality Team - WHAT WE ARE MEASURING WOMEN’S EMPOWERMENT  These Effectiveness Reviews seek to understand the extent to which Oxfam projects have contributed to women’s empowerment, broadly dened as a process whereby a woman is transformed from a situation with limited power to a situation where power is enhanced. Oxfam recognises women’s empowerment to be a complex, multi-dimensional concept. While not arguing for a standard set of women’s empowerment indicators that are applicable to all contexts, Oxfam has developed a multi-dimensional index to support measurement of this hard-to-measure area that sets out ve dimensions of women’s empowerment that the organisation considers to be important in all contexts. The framework is structured as follows: ã Personal  – this level of change includes characteristics of empowerment taking place within the person. It is divided into two dimensions: o Power from within : o Power to : ã Relational  – this level of change includes characteristics of change taking place in the power relations of the woman’s network. It is divided into two dimensions: o Power with : o Power over  : ã Environmental   – this level of change includes measure of change taking place in the broader environment. While the framework is xed, context specic indicators are developed under these dimensions in a participatory manner with local Oxfam staff, partners and communities in order to capture what women’s empowerment means in the particular socio-economic contexts under analysis. These are then combined into a composite index, which provides an overall measure for empowerment. The framework recognises three possible levels where change can take place: at individual level; in the relationship with others; and changes in the broader environmental. These levels of change are then combined within a structure of power (within, to, with, and over).In cases where they are not directly included in the overall measure of empowerment, Effectiveness Reviews selected under Women’s Empowerment also seek to investigate unexpected project effects such as incidences of violence against women or unpaid care work. A specic module has been developed to measure the latter, which includes how time is used, the intensity of care work and the redistribution of activities or responsibilities within the household. This dimension includes measures of change in personal self-condence and self-esteem, personal opinions, personal attitudes and believes.This dimension includes measures of changes in personal autonomy and individual capability to decide actions and carry them out.This dimension includes measures that recognise empowerment as a collective  process , which requires the support and interaction of other peers and organisations. It therefore includes measures of social capital, and participation in community groups. This dimension measures changes taking place in the power relationship between individuals, including changes both within the household and within the community.   COMMUNITY DEVELOPMENT PROJECTS Safe Age of Marriage and Women’s Economic Empowerment in Yemen, evaluated in 2012/13. Photo: Wolfgang Greesman/OxfamWomen’s economic leadership through horticulture planting-material business, Rwanda. Evaluated in 2013/14. Photo: Simon Rawles/Oxfam
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks