Please download to get full document.

View again

of 30
All materials on our website are shared by users. If you have any questions about copyright issues, please report us to resolve them. We are always happy to assist you.

Seminarie/Séminaire VEP/SWEP

Category:

Poems

Publish on:

Views: 31 | Pages: 30

Extension: PDF | Download: 2

Share
Related documents
Description
Seminarie/Séminaire VEP/SWEP Outcome/impactmeting: wat is de meerwaarde en plaats van kwalitatieve benaderingen? L analyse d impacts/de résultats: quelle est la plus value et la place des approches qualitatives?
Transcript
Seminarie/Séminaire VEP/SWEP Outcome/impactmeting: wat is de meerwaarde en plaats van kwalitatieve benaderingen? L analyse d impacts/de résultats: quelle est la plus value et la place des approches qualitatives? Design Charles & Ray Eames - Hang it all Vitra 29 maart 2011 Agenda van de dag 9.30: Introductie: kwalitatieve benaderingen voor outcome/impactmeting 10.00: Workshops Workshop 1: PADEV (Univ Amsterdam, ACE Europe, ICCO, Woord en Daad) Workshop 2: Sensemaker (Vredeseilanden) Workshop 3: PIA (CordAid) Workshop 4: Quasi-experimentele impactmeting (VVOB) 11.30: Paneldebat 12.30: Sandwichlunch Seminarie VEP/SWEP Kwalitatieve outcome- en impactmeting Design Charles & Ray Eames - Hang it all Vitra Huib Huyse 30 maart 2011 Inhoud Hoge verwachtingen beperkte praktijk Waar begint en eindigt kwalitatieve outcome/ impactmeting? Uitdagingen Veel besproken methodieken Wanneer beter kwalitatief, wanneer kwantitatief? 5 centrale lessen I Hoge verwachtingen Beperkte praktijk Outcome- en impactmeting: the missing link? 2009: steekproef van 40 NGO projecten: geen impactmeting, beperkt info op outcome-niveau : Evaluatie prestaties Belgische bilaterale samenwerking (30+ programma s): 1 impactstudie, beperkte outcome info : Review van 146 evaluatierapporten van NGO onderwijsprogramma s in Nederland: geen enkele impactstudie, beperkt materiaal op impact en outcome-niveau Beweegt er iets in het evaluatielandschap? Nederland: nieuw evaluatiebeleid IOB 2010 With Without Before After VK: Value for Money USAID nieuw evaluatiebeleid 2011 Today, I'm announcing a new evaluation policy that I believe will set a new standard in our field...we got to have study designs that allow you to compare impact against real counterfactuals. And that's at the heart of this. That will allow this to actually - one of our legacy items, we hope here, is sort of funding randomized controlled methodologies at a much higher level Maar vooral RCT en quasi-experimenteel those development programs that are most precisely and easily measured are the least transformational, and those programs that are most transformational are the least measurable (Andrew Natsios, former head of USAID, 2010) The Big Push Back! (Eyben, 2010).. A very real risk that the current evidence-based trend will quash organisations whose work has not yet been or cannot be conveniently evaluated.. (PPV-paper, 2011) Hiërarchie in evidence? Randomized controlled trials Quasi-experimental studies Before-and-after comparison Cross-sectional, random sample studies Process evaluation, formative studies Action research Qualitative case studies and ethnographic research Descriptive guides and examples of good practice Professional and expert opinion User opinion 10 3? Kwantitatieve & kwalitatieve methodieken: 5 klassieke continua Positivist Social constructivist Type of information Numerical Non-numerical Type of population coverage Type of population involvement Type of inference methodology General Passive Deductive Specific Active Inductive Type of value framework money-metric multi-dimensional value II Waar begint en eindigt kwalitatieve outcome/ impactmeting? Monitoring & evaluatie, impact assessment? (Intrac 2010) What? When? Monitoring Evaluation Impact Assessment Measures ongoing activities During implementation 5 OECD DAC criteria: relevance, effectiveness, impact, In middle or end of programme cycle Assesses changes in peoples lifes At any stage, or after end Scope? intervention intervention Affected population Focus? outputs outcomes Impacts Question? Are we doing the thing right? Are we doing the right thing? What has changed for whom, How significant is it for them? Experimentele en quasi-experimentele methodieken Before After With Without Doel: Causale verbanden tussen op voorhand vastgelegde variabelen (context-independent) Generaliseerbare theorieën voor sociale verandering Modellen met voorspellingskracht Alternatieve strategieën voor de selectie van steekproeven en cases (Flyvbjerg, 2011) Random selection Information-oriented selection random sample Extreme/deviant cases stratified sample Maximum variation cases Critical cases PADEV: perceptions of (representatives of) beneficiaries in a location Outcome Mapping: collective review of changes at outcome level CORT: multiple lines and levels of evidence NONIE verklaring ivm impact assessment (2008) NONIE advocates an eclectic and open approach to finding the best methods for the task of impact evaluation drawing on the wide range of techniques available from different disciplines II Uitdagingen Waarom is outcome- en impactmeting zo moeilijk? 1. Meten van sociale verandering Families Banks PHCs 2. Dikwijls veel actoren op verschillende niveaus SHG Police Community Leaders State NGO State NGO State NGO State NGO State NGO State NGO BAIF IDRC CIDA Swayamsiddha 3. De keizer heeft geen kleren (Nigel Simister, 2010) Langs de kant van de back-donors Onrealistische tijdskaders Ask a stupid question... Gevaar van competitie Triomf van accountability boven leren Langs de kant van de ontwikkelingsorganisaties Gebrek aan initiatief toen er ruimte voor was Versoepeling van rapporteringsverplichtingen niet overgedragen verder in de keten Over-simplificatie van de hulp boodschap 4. Verschillende informatienoden Assessment van resultaten (quality of results) Project planning & management Organisational learning Verstaan van en onderhandeling tussen verschillende stakeholder perspectieven Accountability beleidsformulering IV Veel besproken methodieken Welke worden als kwalitatief omschreven? CORT (Jesse Dart) Wat hebben ze gemeenschappelijk? Vertrekken vanuit complexiteitsdenken Niet-experimenteel Gebruik van narratives/verhalen Geen (klassieke) indicatoren Passen eerder binnen interpretatieve paradigma in termen van bijdrage aan dan pure causaliteit verbanden tussen acties en objectieven, niet op basis van context-vrije variabelen Enkele verschillen.. Outcome Mapping Most Significant Change Middelen Technische voorkennis Wie leert er? Medium Laag Programmateam & partners Hoe leert men? Horizontaal Medium Laag Programmateam Vertikaal (Horizontaal) Sensemaker Hoog Hoog Programmateam / beleidsmakers PADEV Medium Medium Opdrachtgevers, lokale gemeenschap CORT Medium Medium Programmateam, lokale gemeenschap Vertikaal Horizontaal & vertikaal Horizontaal & vertikaal Scope? Programma Programma Programma Geografische lokaliteit Programma v Wanneer beter kwalitatief, wanneer kwantitatief? Wanneer beter kwalitatief / kwantitatief? Prowse (2007) Flyvbjerg (2001) Rogers (2009) Dean Karlan (2009) Garbarino & Holland (2009) Quantitative (experimental & quasi experimental) M&E designs What & where questions Capturing states and conditions Achieving breadth Instrumental rationality Simple & complicated programs Causal relationships and proving Most unbiased estimate of impact Also for dynamic /complex progrs Achieving aggregation Ambition to predict relationships Qualitative M&E designs Why & how questions Capturing processes Achieving depth Value rationality Complex programs Contribution and learning When impossible with / without When sensitive topics Probing and explaining Explain contextual differences, analyse poverty as dynamic process Combineren van kwalitatieve en kwantitatieve benaderingen? Integrating for better measurement: Some examples Sequencing for better analysis: Examining, explaining, confirming, refuting, enriching Merging findings for better action: Using social analytical frameworks Maar..in de praktijk is kwalitatief eerder een add-on VI 5 centrale lessen 5 key-lessen bij outcome- & impactmeting 1. Belang van methodologische diversiteit in outcome/impactmeting 2. Context en de evaluatievragen moeten de methode bepalen 3. Wat ontbreekt is de expertise om te weten welke methodiek/ combinatie van methodieken te gebruiken in welke situatie 4. Een duidelijke focus op leren versterkt ook het rekenschap afleggen 5. Wanneer durven we ons te wagen aan echte ex-post outcome/impactmeting? Julie Smit Danku! Workshop 1: Workshop 2: Workshop 3: Workshop 4: PADEV (Univ Amsterdam, ACE Europe, ICCO, Woord en Daad) Sensemaker (Vredeseilanden) PIA (CordAid) Quasi-experimentele impactmeting (VVOB)
We Need Your Support
Thank you for visiting our website and your interest in our free products and services. We are nonprofit website to share and download documents. To the running of this website, we need your help to support us.

Thanks to everyone for your continued support.

No, Thanks