Participatory evaluation of local development

Jela Tvrdonova

The evaluation of rural development policies – in compliance with the EU Common Monitoring and Evaluation Framework is the process of the assessment of the public and private funds intervention according its results, impacts and needs, which shall be satisfied. The evaluation deals with the effectiveness, efficiency and relevance of the intervention logic and with the contribution of strategic goal and specific objectives to the solving of problems and needs of the specific geographic area.

If considering this, the local development strategies in the Leader approach can be seen as the tool for practical application of public policies in rural areas at local and/or sub-regional level. As in the case of Rural Development Programmes supported by EAFRD, they are combining public (EU and national/regional) and private funds, benefiting from genuine partnerships of public and private bodies, the same time acting as tools for coordination of development efforts in addressing needs of certain geographic area. Furthermore similarly to Rural Development Programmes they also contain the strategy intervention logic in which area needs are addressed by three levels of objectives – overall (related to strategy), specific (related to development priorities) and operational (related to measures and actions). In these strategies EU funds are used to satisfy above mentioned rural needs and therefore shall be the subject of evaluation (done independently by external bodies) and self-evaluation (done internally by local action groups).

In spite of similarities in the architecture of local development strategies compare to rural development programmes, still several specificities in their evaluation shall be taken in consideration. Among the most important are seven features of Leader approach presumably contributing to the improvement of governance in rural areas, which shall also be the subject of independent evaluation as well as self-evaluation practice at the level of local action groups.

Both – the critical look at the strategy intervention logic and practice in the implementation of the Leader method, shall be taken in consideration, when introducing and developing the evaluation culture inside Leader groups building awareness and capacities in governing public and private resources in favour of development. This has been recognised very recently also by European Network for Rural Development, which started to focus on development and practical introduction of self-evaluation approach and techniques for local action groups. This shall facilitate understanding on how interventions of local development strategies have helped to address needs of respective rural territories as well as how the local action groups as multi-sector institutions can wisely backstop their implementation.

Evaluation of local development strategies

In the ideal case the local development strategy is designed using the continuous feedback control mechanism, which shall lead into well established intervention logic including setting up the monitoring and evaluation framework as the organic part of the strategy already in the ex-ante phase. Such an approach facilitates later the mid-term and ex-post periodical evaluation, especially if prepared monitoring and evaluation frameworks are used on-goingly during the entire implementation of the strategy.

That means to answer evaluation questions designed against strategy objectives – overall and specific, via continuous data collection for selected indicators starting at the baseline period (the 0 time of the strategy implementation) and baseline value of each selected indicator, which shall be analyzed in situation and SWOT analysis of the strategy.

In the figure below you can see the monitoring and evaluation framework in relation to the strategy intervention logic.

Monitoring and evaluation framework of the strategy

The figure can be explained as follows:

• Hierarchy of objectives of the intervencion – left site of the figure (green color) starts with the strategic goal, down to specific objectives and operational objectives at the level of measures
• Evaluation questions – are questions at the level of particular goals/objectives, which we have to ask in oder to know to which extent the whished changed was achieved due to the strategy. In the framework they are also placed in the hierarchy.
• Indicators – each question shall be answered with the help of indicators – measurable units, which measure changes at the level of goals and objectivs. They also create their own hierarchy. The question facilitate the quality of selection and combination of indicators.
• Baseline situation – are values of selected indicators at the beginning of intervention. The situation analysis is always broader that areas targeted by objectives, therefore we recongnise two types of baseline indicators – those related to objectives – which are also in the framework. The contextual baseline indicators are describing the overall socio-economic situation. For the above framework we use the objective related baselines values of already set up indicators at each intervention level.
• Target levels (benchmarks) – are target values of monitoring and evaluation indicators, set up in line with the whished change, which can help to reflect the evaluation. Actual values of indicators are compared with target levels as well as with baseline levels. The distortion found between actual and target values are subject of evaluation as well.

In brief that means, that mid-term and ex-post evaluation shall be conducted from two points of view:

• Phase 1 – evaluation of the strategy from the point of established intervention logic and how it reflects the given territory, its challenges and needs
• Phase 2 – so called ‘bottom up’ evaluation reflecting the real use of allocated resources by final beneficiaries at the project level and their real contribution to the strategic and specific objectives in the frame of selected development priority areas and their direct and indirect effects

Evaluation of local action groups

Next to the strategy intervention logic and final effects evaluation, the equally important is to look at the local action group itself as an institution, or/and administrative unit back stopping its implementation. The task of this part of evaluation is to assess if local action groups has sufficient decision making mechanisms, which respect Leader principles, application and selection procedures, monitoring and evaluation capacities and if it has flexible, effective and efficient management structures, is open to self-assessment practice and if it is utilise the lessons learned from the implementations for future governance of rural areas using integrated manner.

In this context the following activity areas shall be examined if evaluating institutional and administrative background:

• • Organizational structure of local action group and its operational ability
• • Management structures and their operational ability
• • Internal decision making processes –at the strategy and project level
• • Safe guarding of strategy and project implementation
• • Safeguarding of the strategy monitoring and evaluation system

In the area of organisational structure of the local action group and its operational ability the following partial areas shall be examine:

•  If the representation in side of partnership is equally split up in term of public, private and civil sector
•  If partnership contains balanced geographic, social and gender representation
•  If the structure of executive bodies enables effective implementation of the strategy (mainly the project cycle)
•  If the decision making structures are representing the composition (membership structure) of the local action group
•  If the selection committee covers professionally the development priority areas of the strategy
•  If the structure of LAG enables the effective monitoring and evaluation of the implementation

In the area of managements it is important to examine:

•  If the decision making power between LAG and management are split effectively
•  If the management has enough human capacities for the implementation of the strategy
•  If the management has enough professional capacities tp help final beneficiaries to help final beneficiaries in project writing
•  If the management is well prepared technically and financially
•  If the management is recognised in the territory and is accessible for beneficiaries
•  If the management has ability to create contacts and networks among parties involved inside and outside of the LAG

In the area of strategy and projects decision making process shall be examine following:

 If the strategy and its priorities is prepared the“ bottom up“ way

 If the strategy and its priorities is approved the“ bottom up“ way

 If projects are selected and approved in line with the strategy and its priorities

 If approved projects are spread around the territory – (territorial gaps)

 If approved projects are innovative and bring new solutions for the territory

 If approved projects have high added value for the territory

In the area of safeguarding of the strategy and projects implementation to examine whether:

 The high independence in financial decision making of the LAG exists in the system

 The LAG has enough capacities for identification, selection and control in the process of project implementation

 The clear rules in project implementation exist

 Public was informed sufficiently with the implementation rules

 The sufficient amount of eligible beneficiaries for planned measures exist

 Eligible beneficiaries have enough ability to co-finance projects

In the area of monitoring and evaluation, whether:

 LAG has sufficient human capacities for monitoring of local development strategy

 The management of LAG has enough human capacities for monitoring of projects

 The LAG has sufficient human capacities for the strategy evaluation

 The LAG has created the system of regular self-evaluation of the strategy

Practical experiences with using evaluation techniques in updating local development strategies in Slovak and Czech Republic and their applicability in other MS

The above mentioned approach has been used in evaluating of local development strategies and local action groups – both in Czech ( MAS Cesky Zapad, MAS Kyjovske Slovacko, MAS Straznicko, MAS Frydlandsko), and Slovak Republic (MAS Radosinka, MAS LEV, MAS Malohont, MAS Dolná Nitra, MAS Požitavie Širočina, MAS Vršatec) in mid-term period of the implementation of Leader Axis (Axis 4 of Rural development programmes – 2007- 2013).

The method was possible to apply across a quite broad spectrum of local action groups in both countries and it can be applicable in other MS as well. In all cases evaluation results have served as the important inputs for the mid-term up date of strategies – objectives, measures and financial allocations as well as feedback for local action groups in identification strengths and weaknesses in practical application of seven features of Leader approach in respective LAG territories.

As the first step the deep look into the relevance of the selected intervention logic was taken. Both – evaluators and members of local action groups analysed if defined objectives, selected measures/actions and allocated finances were still appropriate within the development context and if the strategy was properly followed via implemented projects. Second – the institutional capacity, including management, implementation and monitoring capacity, was assessed. Data were collected either from financial and monitoring tables or via interviews and focus groups sessions. No sophisticated econometric methods have been used but rather the approach to understand what is happening in the territory due to the strategy and if all resources were utilised effectively and efficiently in addressing defined needs.

The following lessons can be drawn based on evaluation results mentioned above:

 Continuing application of ‘linear strategic planning’ based on experiences from socio-economic planning at the village level, which does not respect the intervention logic and several times only represent wish list instead workable intervention logic (e.g. fails and gaps in intervention logic – implemented measures have not supported achievement of strategy objectives.)

 Failures in SWOT, which has not analysed the situation properly and therefore could not been used as the justification for objectives, measures and actions.

 Missing problem analysis and selection of development priorities according wishes of LAG members

 Weak formulation of objectives and priorities, which did not follow SMART approach (simple, measurable, achievable, realistic and timely) and caused difficulties in evaluation. Often objectives have not followed the hierarchy of intervention (starting with horizontal/area based objectives and coming down to operational objectives at measure/action level.

 Missing time frame for intervention. Often strategies were limited in time, which caused difficulties in setting up financial allocations and designing actions and limited evaluation

 Missing financial planning skills, which has caused unrealistic expectations, or support of activities which did not support the achievement of objectives

 Missing monitoring and evaluation framework – mainly missing indicators, which could help to design strategy better already in programming stage.

 Project selection criteria not corresponding with strategy objectives, which enabled to select projects which were not in line with strategy objectives

 Missing management, networking capacities (both in the territory and outside)

The literature

Commission Regulation 1974/2006 lying down the detailed application of the EC Regulation 1698/2005

EC Regulation1698/2005 on support for rural development by EAFRD

Leader approach – the basic guide, EC Fact sheets, 2006

TEPA team of authors: Animating local partnerships in rural areas – a practical guide, 2008

The Common Monitoring and Evaluation Framework for the EU Rural Development Programs, Handbook and Annexes – A, B, C, E, 2006

Tvrdonova J. The evaluation of local development strategies of Leader type and their institutional backstopping, UZEI, Prague, 2009

 

March 2012