How can civil servants measure the impact of their policies and programmes? To help public managers answer this question, EIPA Barcelona has been organizing seminars on Capacity Building in Impact Evaluation: Counterfactual and theory-based approaches for more than ten years. The last session took place on 22-23 November 2018 in Barcelona with 17 participants coming from 9 different countries and 3 European agencies.
Read a short recap of our expert Garcia Arribas on how this seminar went.
Impact evaluation and how it is performed in the EU institutions
On the first day, I introduced the subject to our participants by underlying the importance of impact evaluation and how it is performed in the EU institutions, especially at the European Commission. Then, Bradford Roehmer (Oxford Research) started his presentation on theory-based approach.
This analysis is particularly set to open the black box of causal mechanisms and understand how and why a policy or programme worked in practice. His presentation provided several practical examples coming from his own experience on evaluation.
Bradford also showed participants what good and bad practices are in performing process evaluation. In particular, he explained how too many details may lead to a lack of clarity and lose its informative purpose on the mechanism the evaluator wishes to explain. His presentation clarified terminologies used in evaluations to ensure a structured analysis and highlighted points to be taken in consideration in practice (for instance unintended consequences).
Following the lecture, the day concluded with a practical session when participants worked in groups to find a way evaluate a programme using the theory-based approach.
Counterfactual approach of policy evaluation
On the second day, Bradford Roehmer presented the counterfactual approach of policy evaluation. It aims to quantify the effect of a programme or policy.
The quantitative approach is most fit when large data is available to the evaluator. It covered Randomized Field Trials and quasi-experimental schemes. As in the first day, participants were invited to use their knowledge in practical sessions.
Essential exchange with other participants
During our seminars, not everything is about lectures. Exchange with other participants is essential. Participants also enjoyed times together during lunches and at a welcome dinner in a Catalan restaurant. This gave ample opportunity to ask questions in a less formal manner and to listen to each others’ experiences. Have a look in this one-minute video for a short impression on this seminar!
The views expressed in this blog are those of the authors and not necessarily those of EIPA.
