Evaluating rural development: Mediating social learning within LEADER

By:               Chris High       (Open University – Systems Department)

                     United Kingdom,   c.high@open.ac.uk

                     Gusztáv Nemes          (Hungarian Academy of Sciences –        Institute of Economics) Hungary,   nemes@econ.core.hu

                     ESRS XXI Congress, WG12 – Keszthely, 2005 August

Introduction

In this paper, we discuss the relevance of different understandings of knowledge to rural development by looking at how rural development is evaluated in Europe.  After outlining some of the links between knowledge and evaluation, we turn to a specific area of development policy in the EU that we feel requires more attention, and in which knowledge is a central issue:  How to evaluate the LEADER programme in a way that resolves the conflicting learning needs of the various stakeholders concerned.  We focus on the LEADER European Community initiative, because it represents a progressive trend within European policy in terms of stakeholder engagement, social learning and systemic methodology for addressing rural needs.

Within LEADER, systemic evaluation could potentially engage with a wide range of perspectives in rural development, across different scales of governance and national and regional contexts, but in practice the articulation between these levels is weak. Of central importance within this is the contradiction between the formal, canonical knowledge required by the European Commission and national governments, and the embedded, informal learning systems that are at the heart of LEADER.  These connections are highlighted through an examination of the literature on participatory development, endogenous development, and governance.

Taking the relationships between different knowledge systems at different scales of governance as our theme, we turn to some of our previous research to highlight some important themes that help to illuminate the relationship between formal institutions and the informal communities of place and practice that give rural development its social dynamism.  What these studies share is a concern with the experiences and developmental paths of rural areas from a micro-perspective, and the relationship between micro and macro structures of governance. Focusing on knowledge and learning, we suggest that (i) systemic (High, 2002) or social learning (Ison et al, 2004; High, 2005), (ii) reflexive agency (Nemes, 2004a; Nemes et al, accepted) and (iii) shadow networks (High et al, 2004c; Pelling & High, 2005), collectively offer some directions for developing an alternative evaluation strategy for LEADER+ and its successors.

Knowledge and Evaluation in rural development

Evaluation of rural development need not be confined in scope to formal projects and initiatives, but for the purposes of this paper we shall restrict our discussion to the case of formal development schemes.  In terms of evaluating initiatives such as projects and programmes, evaluation has been defined as “…a periodic assessment of the relevant performance, efficiency and impact of the project in the context of its stated objectives” (Casley & Kumar, 1988: 12).  It centres not only on a forensic appreciation of the consequences of a particular approach, but is intended to improve future decision making and planning (Jones, 1982: 7). 

In general, such evaluation concerns that formation of judgements about the course and impact of initiatives in respect of the values that underpin them.  The two main questions are “Did it work?”, and “How might we do better?” and both these questions are fundamentally value-laden.   That is on the surface evaluation may seem to be about judgements of fact, but these are almost always intertwined with judgements of value (Checkland & Casar, 1986). Where multiple stakeholders are present, goals, purposes and understandings often diverge, even where different parties have been able to co-operate – because as Checkland (1999[1981]; 2000) makes clear, consensus (the complete alignment of goals and values) is a special case of accommodation (where enough alignment occurs for progress to be negotiated). Thus even agreeing on the details of what happened, never mind the results, is fraught with difficulties generated by divergences in standpoint. 

This is problematic, because the basic metaphor of much evaluation practice is one of measurement, and in practice, conventional methods of evaluation are based on the experimental methods of the natural sciences (Mtshali, 2000: 69).  This lends itself to an approach which emphasises financial and other quantitative metrics, though qualitative methods based on indicators are becoming more common (Department of the Environment, 1996; Goyder et al, 1998).  Other qualitative methods that may help capture aspects of experience that can fail to be identified through pre-coded surveys, include  focus group discussion, case study, and semi-structured interviewing. (Baker & Schuler, 2004). However, this does not evade the issue that projects, programmes and other initiatives are contextually sensitive.  Different stakeholders are entrenched in power figurations with intertwining relations of dependency and accountability, making it difficult to argue for rational, value-free judgements on the effects of a project and the lessons that need to be carried forward. This is most critical in the case of evaluating social impact, where understandings of impact are intrinsically socially constructed.

We suggest that part of the difficulty is a limited understanding of what knowledge comprises. On the one hand, there is a dominant cultural metaphor of knowledge as a commodity, based on its transferability. This rests on an amalgamation of knowledge and information – knowledge as data, representations of experience, and in particular to a mode of human communication which appeals to a shared, concrete reality. Academically, this view rests on work on communication in terms of coding, signalling and transfer -  most clearly expressed by Shannon & Weaver (1949 cited in Richardson 2004).  Here, information is a commodity to pass around, and where doubt arises it is a matter of interpretation which can be cleared up through reference to a shared external reality.

On the other hand, as far as we are concerned, knowledge stands for a much more socially textured understanding of human understanding.  Knowledge is constructed, but this indicates an intersubjective reading of the process, where knowledge arises within the ongoing interactions of social beings.  This lies in between the traps ingrained in both the realist and the subjectivist positions (Maturana & Varela, 1992), a radical constructivism that has much in common with approaches such as situated learning (Lave & Wenger, 1991; Wenger, 1999) and social learning (Finger & Verlaan, 1995; Leeuwis & Pyburn, 2002; Ison et al, 2004; Keen et al, 2005a). 

This view of knowledge is much closer to Krippendorf’s (1993) metaphor of knowledge as a dance, where knowledge unfolds within ongoing, social relationships. For Ison et al (2004: 13), for example, knowing “…occurs with the act, the process, of constructing an issue and seeking improvements.”   Thus knowledge is embedded (sensu Granovetter, 1985), particular to each social (and hence spatial) environment which is being considered.  Realist assumptions that there are robust procedures for validating knowledge claims based on comparing them with an external reality break down when it is understood that that procedures through which we might do so are themselves socially constructed. The multiplicity of knowledge is therefore a key issue in the evaluation of rural development, and opens up questions of who learns what.  Different responses to this question lead to quite different forms of evaluation. 

The evaluation of multi-stakeholder initiatives carries us into a territory where knowledge is sociologically very interesting and also has profound practical consequences.  What is understood about a situation configures purposeful action within it and beyond it.  Evaluation potentially forms an important part of creating understandings about situations because it is an opportunity for different stakeholders to surface and negotiate judgements of fact and value. However, it is not only important what we know about a situation, but how we know it, and what we recognise as knowledge is profoundly political.  As Scheurich (1997) puts it, ‘…epistemological enactments are ultimately political or ethical enactments.’  This is certainly the case with EU’s LEADER rural development programme, which is planned and delivered along very different lines to much other EU funding.  The difficulty is that the evaluation of LEADER appears to be working under a different epistemic logic than its operation.

Understanding LEADER[1]

LEADER is a strand of European Union rural development funding, with the current iteration LEADER+, following on from LEADER I and LEADER II.  LEADER, an acronym derived from the French for ‘links between actions for the development of the rural economy’ has promoted rural development in territories across Europe since 1991 through funding programmes based on consultation with local action groups, which can comprise representation from local government, local businesses and local civil society.  The intention is for innovative rural development actions to emerge that valorise indigenous financial and cultural resources to produce sustainable development.  In many senses LEADER is a cutting edge programme, progressive in intent and execution, and much admired and imitated for its ability to deliver heterogenous rural development plans which draw in multiple levels of governance in the service of local development priorities (Saraceno, 1999: 439).

Saraceno’s  (1999) summary of LEADER’s singular features includes that it: (i) is a locally based approach, (ii) is bottom up, (iii) incorporates the presence of a local action group, comprising local public agencies and/or local enterprises and/or by local private resi­dents, (iv) emphasises innovative actions, (v) is integrated rather than sectoral, (vi) builds connections through networking activities, and (vii) gives much freedom to local groups in terms of allocating spending.

Of these, we want to highlight two features in particular.  The first is the experimental nature of LEADER projects.  The intention behind LEADER was to have:

 “…a kind of showcase for what we are trying to encourage on a larger scale in the mainstream rural development programmes; the emphasis of the new initiative (i.e. ‘LEADER Plus’) should therefore be on supporting pilot rural activities …(it) must be a laboratory for rural development to encourage the emergence and testing of integrated and sustainable development approaches.”

F. Fischler – European Commissioner for Agriculture in 1998, quoted in Mosely (2000: 111)

This is not always the case in practice, and we are certainly aware of LEADER initiatives that are based on very mature applications of local resources, that is, where LEADER has become just another pot of funding to draw down for local communities to sustain their on-going activities.  We are not aware of any literature on subsidy-farming in rural development, but given the increasing prevalence of structural funding in Europe and elsewhere it would make an interesting study.  The main point however, is that given that LEADER is supposed to be about learning from innovation, there is a clear argument for making learning a central tenet of the approach, and therefore evaluation should have a high profile and be well integrated into LEADER programmes and projects rather than a bolt-on activity.

The second important feature for the purpose of this paper is the participatory nature of LEADER.  That is, the engagement of local stakeholders in the formulation and delivery of programmes and projects, the focus on local resources and the recognition of different cultural and institutional contexts clearly marks LEADER as a member of a broad school of social and policy thinking linked to cognate terms such as citizenship, participation, social learning, customer focus, governance and endogenous development.  As with many interesting concepts, there is no universally agreed definition and we do not suggest there should be. 

Participation, for example, has an expansive literature and can refer to a correspondingly broad range of practices, but the core of it is that the planning and delivery of projects and services should include those stakeholders whose livelihoods are most directly affected by them.  This clearly includes but is not limited to potential beneficiaries, and many participatory analyses focus on those who may experience detrimental outcomes.  Participation as a policy approach has a particularly strong representation within thinking and practice in developing country contexts, where it became a favoured mode of development assistance by international aid agencies from the 1980s onwards.  There is a robust relationship between participatory practice and the rise of participatory modes of research and action (Berardi, 2002), although recently there has been extensive academic debate about the path participatory practice has taken as its application scales up (Cooke & Kothari, 2002; Hickey & Mohan, 2005).  In terms of knowledge, one of the main traps highlighted is the assumption that an internal/external dichotomy of knowledge is the whole picture, when in fact much ‘local knowledge’ represents a complex mosaic of different standpoints (Goebel, 1998; Guijt & Kaul Shah, 1998).  Participatory is not straight-forward, and participatory decision-making can favour non-marginal groups where society is stratified (Gupte, 2003).

In European discourse on rural development, many of the same themes as participatory development in Africa, Asia and Latin America are reprised in discussions of endogenous development (Bassand et al, 1986; Van der Ploeg et al, 2000; Nemes, 2004a).  The notion of endogenous development, as suggested by Bassand et al. (1986) has been promoted in opposition to more ‘modernist’ notions of development.  It is perhaps best understood as “the  hypothesis that improvements in the socio-economic well being of disadvantaged areas can best be brought about by recognising and animating the collective resources of the territory itself” (Nemes, 2004a).  The concept of endogenous development is closely aligned with a developmental ethic that shifts attention from sectoral to territorial logics of change (Ray, 1999b) and recognises the role of a multi-functional agriculture in the wider rural economy and landscape (OECD, 2001; Wilson, 2001).

Saraceno (1999), however claims that the literature is strongest on endogenous development as a more or less spontaneous process, and that there is less material on how these observed characteristics may be turned into policy prescriptions which can then be implemented.  This perhaps reflects Bryden’s (2000) observation that in practice any idea of a new rural policy in Europe trails well behind the rhetoric. For Nemes (2004a), the main problem is the inability of the European centre to simultaneously enact its rhetoric on endogenous development and local participation, and also comply with its own rules on accountability and transparency for public spending.  In another context, Ison et al (2004) observe the same phenomenon with the European Water Framework Directive, where they argue that formal institutions are a disabling factor for the social learning approach promoted by the directive.

In response to this, we note that there may well be existing research which addresses these problems in the international development literature, and in particular a comparative review of work on endogenous and participatory development with a focus on practice would be timely, although daunting.  In response to Saraceno in particular, we would merely point out that ‘policy prescriptions’ is perhaps the wrong metaphor for the relationship between policy and endogenous development. This relationship is by no means straight-forward (see Rengasamy et al, 2001; Vorley, 2002), but there is much interesting research coming forward that suggests how it may be elucidated.  An important factor to take into account is that the professional skills and expertise that are required within many governmental agencies and organisations are completely at odds with those required for endogenous development.  Working in partnership, across institutional boundaries (Williams, 2002), requires a shift in emphasis from management to facilitation, and from ‘expertise on top’ to ‘expertise on tap’ (Gibson, 1996; Chambers, 1997; High, 2005: 139).  Does Europe lag behind other parts of the world in this respect?

Another broader literature in which these issues are discussed is that concerned with a general trend from government to governance (Richards & Smith, 2002; High et al, accepted). Here, government is associated with making and enforcing decisions through centralised control and hierarchical structures.  Governance, on the other hand, is associated with networks and institutional arrangements that reflect more horizontal structures and less centralised power, with an increased role for non-governmental actors in public policy formation and delivery.  According to Stoker (1998) the objectives of governance and government are the same: they are concerned with creating the conditions for collective action and public order.  The difference arises through their respective processes, and under governance the role of the state shifts from one of control to one of co-ordination, using new mechanisms to guide a plurality of network actors (Stoker, 1998; Bach & Flinders, 2004).

The significance of the participatory nature of LEADER as a process of governance is that classically, participation highlights the tensions between local and external actors in development activities, challenging notions of power and control (Webber, 1994: 111-2; Pretty et al, 1995; Blackburn & Holland, 1998: 192).  It is not that these tensions are not present in non-participatory forms of governance, but rather that the contradictions between endogenous and exogenous control is highlighted.  The paradox is that when delivery across a spectrum of projects, localities and cultural contexts becomes important, the very localised particularities upon which the success of participatory initiatives depend may be suppressed by the structural forces which hold wide-scale programmes together. While a small-scale pilot study or experimental project can go astray without much recrimination, there is considerable force in demands for public accountability when an approach such as LEADER begins to consume greater sums of public funds, as it has during each incarnation.

In summary, LEADER represents a fine example of multi-level governance - a complex  process involving the interaction of multiple stake-holders often with different definitions of ‘the problem’, working at different political levels (Murphy & Chataway, 2005).  If it is to become a process of integrated rural development (Nemes, 2004a; Nemes et al, accepted), where endogenous and exogenous governance structures work together to promote sustainable rural development, then appropriate institutional capacity within local territories and governance relations which are supportive of subsidiarity and devolution are required.  Given the emphasis on learning, it behoves LEADER to adopt a participatory approach to evaluation. As we shall discuss in the next section, although there have been efforts in this direction there are still significant differences between the relatively successful implementation of a participatory approach in planning and delivering LEADER projects, and the institutional framework under which it is evaluated as a whole.

Evaluating LEADER+: The challenge

The evaluation of LEADER+, the current incarnation of LEADER, is subject to European Commission guidelines (DGA, 1999) for all rural development initiatives supported through the European Agricultural Guidance and Guarantee Fund.  The purpose of this is two-fold:

“Rural development evaluation must provide information on the implementation and impacts of the co-financed programmes. The aims are, on the one hand,  to increase the accountability and transparency with regard to the legal and budget authorities and the public and, on the other hand, to improve the implementation of the programmes by contributing to informed planning and decisions concerning needs, delivery mechanisms and resource allocation.

From the introduction to “Guidelines to the evaluation of rural development programmes (DGA, 1999: 4)

Evaluation is seen as an important part of co-ordinating rural development across the EU, helping to relate aspects of development initiatives to the general objectives of rural development set at European level in terms of relevance, effectiveness, efficiency, utility and sustainability.  In addition, LEADER+ (the LEADER programme running from 2000-2006) is subject to further regulations of its own, the “Guidelines for the evaluation of LEADER+ programmes” (European Commission, 2002). 

These build on work for the LEADER observatory during LEADER II, and are intended to address some of the issues identified as problematic during earlier manifestations of LEADER.  In particular, they are intended to demonstrate the “added value” of LEADER+ – the utility of the distinctive features of LEADER, as discussed above.  This is essentially a political project, something that is more than hinted at in the documentation, and in associated material (LEADER European Observatory, 1999) – a primary motivation for the particular approach to evaluation adopted at the European Commission level is to demonstrate the LEADER provides value for money, compared to other sources of funding for rural development.

This requires an understanding of the operation of LEADER in each local context, something which it is argued that conventional European rural development evaluation fails to do (Saraceno, 1999). The LEADER+ evaluation guidelines (European Commission, 2002), therefore place an emphasis on qualitative evaluation: a framework of common evaluation questions that each local LEADER+ programme is obliged to address, relating to the overall objectives of LEADER, and guidance on the formulation of programme-specific evaluation questions, which depend on the features that individual LEADER+ programme has emphasised.

This goes some of the way towards the challenges that Saraceno (1999) suggests are necessary for the effective evaluation of LEADER: 

(i)                 How to define and assess the distinctive features of LEADER separately?

(ii)               How to relate these to development outcomes?

(iii)             How to aggregate LEADER outcomes at national and European level?

(iv)              How to produce relevant evaluation information for different governmental stakeholders?

However, while these are important questions, and ones that we do not necessarily have any better answers to than Saraceno or the authors of the LEADER+ evaluation guidelines, we suggest that if LEADER evaluation is to be an effective tool for learning, then it needs to change.  The current system is still top-down:

“Precisely because there has been a decentralization of decision making in favour of the local level, it is legitimate that providers of funding would want to know what has been achieved by each group aggregating evaluation information at regional, national or European levels. This, or any other information needs that may be established, requires common questions and issues that allow classification of responses into common categories which can then be com­pared. The issue is not so much to find appropriate indicators in this aggre­gation exercise, but rather, as with statistics, to find the appropriate and meaningful classificatory variables and categories of analysis to understand what has been accomplished.”

(European Commission, 2002: 8)

This need not be the case - there are well-established alternative traditions of evaluation that “…involves the stakeholders and beneficiaries of a programme or project in the collective examination and assessment of that programme or project. It is people centred: project stakeholders and beneficiaries are the key actors of the evaluation process and not the mere objects of the evaluation” (OESP, 1997).  This participatory style of evaluation harks back to the 1970s (Wadsworth, 2001: 45), and seeks to appreciate and integrate the knowledge of both insiders and outsiders (Davis-Case, 1990), and professionals and beneficiaries (Wadsworth, 2001). 

The relationships between different knowledge systems within different participatory methodologies form a spectrum between those where efforts are made to include a wider range of stakeholders (ibid: 46) and those where evaluation is led by lay stakeholders who take a substantive part in designing and enacting opportunities for project actors, funders and gatekeepers to learn about the project (Davis-Case, 1990).  The aim is usually to avoid “…the negative connotations often associated with evaluation is that it is something done to people”  (Patton, 1990: 129).  The claim is that evaluation in these terms adds value to development effort through empowering project stakeholders – often a central aim of development efforts. 

In the case of LEADER, there is a stated ambition not to preclude a “bottom-up, integrated approach” to evaluating the impact of LEADER+.  But in practice, we feel this is not the case.  In our experience of LEADER groups, centrally mandated evaluation not only becomes a task quite divorced from the reality of LEADER delivery, but it taints evaluation as something onerous, and directs resources away from delivery. By failing to institutionalise participatory evaluation into the current LEADER+ evaluation arrangements, the evaluation process is out of step with the way that LEADER+ is planned and delivered.  Furthermore, if we are right, there is also serious missed opportunity to build local capacity for social learning through LEADER’s inability to build participatory evaluation in to the system.

It seems clear to us that there are fair and honest reasons why by the third generation of the programme, participatory evaluation is still not institutionalised within LEADER. Within the literature on participatory evaluation, there is a well recognised tension between participatory values and the norms of accountability and power that come with externally driven initiatives.  White (2003: 331-2), for example, illustrates the problem quite clearly: “Inclusion of plans for community participation is a virtual requirement in project funding  proposals of most major donors.  Yet, project evaluation procedures of the same donors may ensure that non-participatory elements of projects take precedence, because their benefits occur within a shorter timeframe and often are more easily quantified for reporting purposes.” The issues become even more complex when evaluation occurs across multiple sites, where the development agenda has been set locally, but the evaluation criteria are set globally (Lawrenz & Huffman, 2003). 

In the European context, the global, political aim of defending the LEADER approach is a necessary part of LEADER evaluation, but is it necessary that this should be at the expense of evaluation being useful?  Not according to Saraceno (1999):  “The often-mentioned conflict between participatory self-evaluations at local level and external evaluations is greatly reduced when we enlarge the scope of the evaluation exercise to include all the stakeholders in the initiative, EU included. We found a great deal of complementarity between the different methods of evaluation. Those groups which voluntarily and periodically under-took evaluation exercises were also those that provided the best and most reli­able information to external evaluators.” 

If it is possible to evaluate federal programmes in the United States (Lawrenz & Huffman, 2003), and international UN (OESP, 1997) and World Bank (Baker & Schuler, 2004) projects using participatory projects, then this suggests to us that at the very least this is a good research topic for rural development in Europe.  Nor is evaluating innovative pilot projects without precendent (Sanderson, 2002). Given LEADER’s progressive profile, there is every chance that success in developing an evaluation system that integrates different forms of knowledge production would be an international model for others to follow.  It would represent an exemplar of social learning in the sense defined by Keen et al (2005b: 4): “Social learning is the collective action and reflection that occurs among different individuals and groups as they work to improve the management of human and environmental interrelations.” 

Thus in addition to Saraceno’s four challenges for LEADER evaluation (above), we would add the following two:

(v)                How to evaluate LEADER in a way that respects the diversity of local knowledge, in accordance with the aims and approach of the programme?

(vi)              How to institutionalise evaluation in LEADER, so as to build local capacity for social learning and rural development?

The biggest challenge we can see in this is how to achieve an articulation between different forms of knowledge, working at different scales of governance.  In the following section, we shall briefly discuss some results from previous research projects that suggest how this might or should happen.

Learning from experience: Knowledge in rural development

Learning and knowledge are important themes in rural development, not least because there is frequently an understanding that things could or should be different, and that in order for this to be so, someone somewhere has to learn to do something differently. The initial impetus for this paper, and the research that we hope will come out of it, is a growing feeling that we have been researching the same issues for many years, albeit in different contexts, and drawing on different traditions of scholarship.  What we have in common is a feeling that thinking about the ‘someone somewhere’, has to take into account the aspirations and capabilities of rural stakeholders and that all too often the organisations that give rural governance its structure and direction fail to do so in important ways.  In other words, we accept that knowledge is an important part of rural development, and find that the plurality of knowledge in rural development is a non-trivial issue.  What we have in common is that we feel we can begin to see ways beyond that.  In this section, we present just the relevant findings from three sets of research ‘in the South’.  All three take the relationship between different levels of rural governance and the processes of communication, learning and co-ordination between as central themes.  They are merely summarised here, but references to more extensive case study material is included in each description.

Systems approaches to learning in Tamilnadu – South India

This research arose in partnership between a PhD student (High, 2002), and members of an Indian NGO, engaged as partners in a multi-country research project into the links between policy and sustainable agriculture (Vorley, 2002).  The NGO, the Society for the People’s Education and Economic Change (SPEECH) were an early adopter of the PRA (Participatory Research and Action) methodology in India, primarily interested in the development of poor and deprived communities in a rural area of South India. The PhD work concerned the development of methodological principles for working with learning and change in sustainable development.

At the time of SPEECH’s founding in the late 1980s, the most local tier of government had been suspended in the state of Tamilnadu, and there was no formal system of local representation (High & Rengasamy, 2002).  SPEECH’s early work consisted of consciousness-raising and non-formal education, as well as support for the creation of representative structures within local communities. As representative local democracy revived during the 1990s, the organisation shifted its focus to sectoral programmes and participatory development, but the organisation’s role as a mediator between local communities and local and central government remained.  This is particularly important because of the complexity of rural development policy, and the lack of engagement between local farmers and local officials in an area where agrarian livelihoods are still critically important.

Details on SPEECH’s mediation strategies and some examples of successful reconciliation between sustainable agriculture and rural policy are reported in Rengasamy et al (2001), summarised in High & Rengasamy (2002) and compared to other rural development arenas in Vorley (2002).  High (2002) additionally reports on action research on learning and communication to report on the Indian case study of PTW to it’s international sponsors, and pilot research to improve the local dissemination of the results in comparison to simply translating the report into Tamil.

In terms of knowledge and evaluation, the main relevant themes are:

• The relevance of indigenous knowledge to rural development, and the analytical strength of approaches that integrate insider and outsider knowledge.
• The importance of communities of practice (sensu Wenger, 1999) in knowledge and learning, and the need for communication strategies that are designed to take the learning processes that link these together into account.
• The need to consider learning and communication systemically, paying attention to emergent effects as different knowledge communities interact, and the utility of systems approaches for doing so.
• The existence and relevance of alternative relational networks that inter-penetrate official structures of governance, and the potential for strategies that successfully articulate between the formal and informal learning systems that drive rural development (High et al, 2004c; High et al, accepted).
• Methodologically, approaches for stakeholder analysis and participation based on invitation were developed and applied.

Integrated rural development in South Transdanubia – Hungary

This research concerns the course of the EU-funded PHARE (Poland Hungary Assistance for the Reconstruction of the Economy) Pilot Action Fund in South Transdanubia, Hungary.  In Hungary during the 1990s, PHARE as a whole became an important driver of rural development, alongside domestic programmes.  But while much physical infrastructure was improved as a result of the EU assistance, PHARE in general was judged ineffective because it was relatively inflexible and wrapped in red tape.  The operation of the PHARE pilot programmes in South Transdanubia was an exception to this rule, and PhD research by Nemes (2004a) investigates why and how this was possible.  The research fieldwork comprised 6 months of participant observation, supplemented by 107 interviews, including members of the regional development council, mayors, local entrepreneurs, national politicians and bureaucrats, and representatives of trade unions, local associations, economic interest groups and tourism organisations.

In Nemes (2004b) and Nemes et al (accepted), the role of South Transdanubian Regional Development Agency, which was created in order to co-ordinate the PHARE pilots in South Transdanubia is highlighted.  Through mix of good contacts, the development of an effective learning organisation, skilful political manoeuvring and sheer hard work, the development agency was able to simultaneously build local capacity for the design and delivery of PHARE projects according to EU norms, and deal skilfully with regional, national and European organisations and institutions to ensure that local needs where adequately taken into account.

In particular the research highlighted the following themes:

• The difficulties that arise from the different institutionalisations of knowledge in rural development at different levels of governance.  In the research these are theorised in terms of the relationship between a central bureaucratic system and a local heuristic system.  When these come into alignment, integrated rural development is possible.
• The role of reflexive agency in rural development, and organisational analogues of reflexive practitioners (sensu Ray, 1999a).  Reflexive agency represents an articulation between individual and organisational learning.
• In particular, reflexive agency is able to bridge the gap between the central bureaucratic system and the local heuristic system, at least temporarily through mediation. 
• Two types of mediation can be distinguished:  Horizontal mediation which operates within a given level of governance to build capacity and forge accommodations, and vertical mediation – a translation process that enables different levels of government to co-operate in spite of having different knowledge frames, and motivations for engaging with rural development.

Adapting to rapid climate change in South Wales - UK

This research took place in the context of a one-year study of adaptive capacity to rapid climate change in the UK rural sector.  Rapid climate change was taken to mean a global cooling induced by the collapse of the North Atlantic circulation, compared to a mainstream gradual warming scenario.  An analysis of past responses to economic, climactic and bio-physical stressors in the sector was combined with literature review on risk management, social and organisational learning, social capital and adaptation to climate change.  The resulting framework was tested in interviews and workshops with local, regional and national stakeholder groups, including farmers, officials and Assembly members of the Welsh Assembly, officials in the Environment Agency Wales and the UK level Environment Agency, and officials in the UK’s Department for Environment, Food and Rural Affairs. The results are reported on-line at www.rcc.rures.net (High & Pelling, 2004; High et al, 2004a; High et al, 2004b), and initial publications include a conference (High et al, 2004c) and a journal paper (Pelling & High, 2005), with more to follow.  Material in High et al (High et al, accepted) also drew indirectly on this research.

The work in Wales highlighted some of the differences in the way that learning and knowledge is institutionalised amongst farmers in South Wales, compared to Welsh officials. Both groups were able to discuss past experiences that demonstrated considerable adaptive capacity: (i) economic stress in the case of the farmers, who were successful dairy farmers at a time of general decline in the UK dairy sector, and (ii) the response to the 2001 outbreak of foot and mouth disease amongst officials, which required a high degree of inter-agency co-ordination amidst considerable uncertainty. In both cases, informal relationships were an important site of adaptive capacity and collective social learning, and the differences lay in the articulation of formal and informal learning systems in each case.

Within this research, the relevant themes were:

• The sophistication of supposedly ‘lay’ knowledge of rapid climate change, and the ability of lay stakeholders to engage quickly and meaningfully with technical and scientific information.
• The significance of ‘shadow systems’ (sensu Shaw, 1997) – informally institutionalised relational networks that interpenetrate formal organisational structures – in organisational and social learning, and hence adaptive capacity.
• The linked nature of formal and informal institutionalisations of knowledge and learning.
• The difficulties that arise when formal institutions operate in such a way as to limit the shadow system.
• The existence of a growing school of thought on organisational learning that suggest ways forward in reconciling formal and informal knowledge structures.

Conclusions: Evaluation and knowledge plurality

In this paper, we have examined some of the problems that arise in evaluating rural development.  In particular, we focussed on the difficulties that have been documented in evaluating the European-wide LEADER stream of development assistance in line with its participatory planning and delivery.  We propose that research into more sophisticated evaluation of LEADER projects and programmes would be timely and would increase the capability of the programme to build capacity for endogenous rural development.  This would however require an approach that is sensitive to divergent stakeholder perceptions and interests, integrating endogenous and exogenous perspectives, and formal and informal learning systems. This would entail a shift from product to process and an emphasis on the capacity of individuals and organisations to deliver this kind of approach.

Understanding that knowledge is plural is central to any improvement of the LEADER evaluation strategy.  A top-heavy, centrally administrated evaluation that privileges scientific knowledge compatible with formal procedures over optional local evaluation and treats knowledge as something that can be passed around like so much lost luggage is not only a missed opportunity at the local level, but for the central level too.  Evaluation that took a social learning approach with the production of knowledge the result of partnership, understood knowledge to be systemic and multi-layered would enrich LEADER evaluation as a whole if the difficulties of aggregation of participatory evaluation could be overcome.  We do not believe this is just a pipe-dream, and feel that our own history of research and experience positions us well to take part in action research on integrating plural knowledge systems based within different levels of governance. The experience of participatory evaluators, particularly in developing countries is encouraging, demonstrating that the integration of scientific and traditional knowledge is not only possible, but enriching (Goma et al, 2001), and that the process effects of participatory learning are ultimately more persistent than the immediate products (1996). 

Reference

Bach, I. & Flinders, M. (2004). Multi-level governance. Oxford, Oxford University Press.

Baker, J. & Schuler, N. (2004). Analyzing urban poverty:A summary of methods and approaches. Working Paper, No. 3399,  World Bank.

Bassand, M., Brugger, E.A., Bryden, J.M., Friedman, J. & Stuckey, B. (1986). Self-Reliant Development in Europe: Theory, Problems, Actions Brookfield, Vermont, Gower.

Berardi, G. (2002). Commentary on the Challenge to Change: Participatory Research and Professional Realities in Society & Natural Resources 15. 847-852.

Blackburn, J. & Holland, J., Eds. (1998). Who changes? Institutionalizing participation in development. London, Intermediate Technology Group.

Bryden, J. (2000). Is there a 'new rural policy'?  International Conference: European rural policy at the crossroads conference. Arkleton Centre, Aberdeen, University of Aberdeen.

Bunch, R. & Lòpez, G. (1996). Soil Recuperation In Central America: Sustaining Innovation After Intervention in Gatekeeper series (55).

Casley, D. & Kumar, K. (1988). The collection, analysis and use of monitoring and evaluation data. Baltimore, John Hopkins.

Chambers, R. (1997). Whose reality counts? Putting the last first. London, Intermediate Technology Publications.

Checkland, P. (1999). Systems thinking, systems practice. Chichester, John Wiley & Sons.

Checkland, P. (2000). The emergent properties of SSM in use: A symposium by reflective practitioners in Systems Practice and Action Research 13(6). 799-821.

Checkland, P. & Casar, A. (1986). Vickers' concept of an appreciative system: A systemic account in Journal of Applied Systems Analysis 13. 3-17.

Cooke, B. & Kothari, U., Eds. (2002). Participation: The new tyranny? London, Zed Books.

Davis-Case, D.A. (1990). The community's toolbox. Rome, Food and Agriculture Organisation.

Department of the Environment (1996). Indicators of sustainable development for the United Kingdom. London, HMSO.

DGA (1999). Evaluation of rural development programmes 2000-2006 supported from the European Agricultural Guidance and Guarantee Fund. Guidelines, No. VI/8865/99- Rev.,  Directorate General for Agriculture, European Commission, Brussels.

European Commission (2002). Guidelines for the evaluation of LEADER+ programmes. Commission Working Document, No. VI/43503/02-REV.1,  European Commission - Agriculture Directorate-General, Brussels.

Finger, M. & Verlaan, P. (1995). Learning our way out: a conceptual framework for social-environmental learning in World Development 23(3). 503-513.

Gibson, T. (1996). The power in our hands. Charlbury, Jon Carpenter Publishing.

Goebel, A. (1998). Process, perception and power: Notes from 'participatory' research in a Zimbabwean resettlement area in Development and Change 29. 277-305.

Goma, H.C., Rahim, K., Nangendo, G., Riley, J. & Stein, A. (2001). Participatory studies for agro-ecosystem evaluation in Agriculture, Ecosystems & Environment 87(2). 179-190.

Goyder, H., Davies, R. & Williamson, W. (1998). Participatory impact assessment: A report on a DfID funded ActionAid research project on methods and indicators for measuring the impact of poverty reduction. Formal report, ActionAid, London.

Granovetter, M. (1985). Economic action and social structure: The problem of embeddedness in The American Journal of Sociology 91(3). 481-510.

Guijt, I. & Kaul Shah, M., Eds. (1998). The myth of community: Gender issues in participatory development. London, ITDG Publishing.

Gupte, M. (2003). Reexamining participatory environmental policy: Social stratification and the gender dimension in Society and Natural Resources 16. 327-334.

Hickey, S. & Mohan, G., Eds. (2005). Participation: from tyranny to transformation? London, Zed Books.

High, C. (2002) Opening spaces for learning: A systems approach to sustainable development PhD thesis, Centre for Complexity and Change,  Open University,  Milton Keynes.

High, C. (2005). Fried eggs and phronesis: ICTs and social learning in rural development  XXI European Congress of Rural Sociology. Keszthely, Hungary.

High, C. & Pelling, M. (2004). Understanding adaptive capacity to rapid climate change: Drawing on theories of social and organisational learning. Rapid climate change project working paper, No. 3,  University of Liverpool, Liverpool.

High, C., Pelling, M. & Dearing, J. (2004a). Adaptation to climate change: Grasshoppers and resilience. Rapid climate change case study, No. 2,  Kings College London, London.

High, C., Pelling, M. & Dearing, J. (2004b). Exploring responses to rapid climate change in Wales. Rapid climate change case study, No. 1,  University of Liverpool, Liverpool.

High, C., Pelling, M. & Rengasamy, S. (2004c). Local agency, adaptation and the shadow system: The institutional architecture of social learning in rural areas of the UK and India. XI World Congress of Rural Sociology, Trondheim, Norway.

High, C. & Rengasamy, S. (2002). Working with the system: Reconciling local activism and the policy process in Southern India. Bridging policy and research case study, No. 13,  Global Development Network, London.

High, C., Slater, R. & Rengasamy, S. (accepted). Are shadows dark? Governance, informal institutions and corruption in rural India in Cheshire, L., Higgins, V. & Lawrence, G., Eds. International Perspectives on Rural Governance: New Power Relations in Rural Economies and Societies.

Ison, R., Steyaert, P., Roggero, P.P., Hubert, B. & Jiggins, J. (2004). Social learning for the integrated management and sustainable use of water at catchment scale SLIM final report, Open University, Milton Keynes.

Keen, M., Brown, V.A. & Dyball, R., Eds. (2005a). Social learning in environmental management: Towards a sustainable future. London, Earthscan.

Keen, M., Brown, V.A. & Dyball, R. (2005b). Social learning: A new approach to environmental management in Keen, M., Brown, V. A. & Dyball, R., Eds. Social learning in environmental management: Towards a sustainable future. London, Earthscan: 3-21.

Krippendorff, K. (1993). Major metaphors of communication and some constructivist relfections on their use in Cybernetics & human knowing 21(1). 3-25.

Lave, J. & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge, Cambridge University Press.

Lawrenz, F. & Huffman, D. (2003). How Can Multi-Site Evaluations be Participatory? in The American Journal of Evaluation 24(4). 471-482(12).

LEADER European Observatory (1999). Assessing the added value of the LEADER approach. "Rural Innovation" Dossier, No. 4,  LEADER European Observatory, Brussels.

Leeuwis, C. & Pyburn, R., Eds. (2002). Wheelbarrows full of frogs: Social learning in rural resource management. Assen, Koninklikje Van Gorcum.

Maturana, H. & Varela, F. (1992). The tree of knowledge: The biological roots of human understanding. Boston, Shambala.

Moseley, M.J. (2000). Innovation and rural development: Some lessons from Britain and Western Europe in Planning Practice & Research 15(1/2). 95-115.

Mtshali, S.M. (2000). Monitoring and evaluation of women's rural development extension services in South Africa in Development Southern Africa 17(1). 65-74.

Murphy, J. & Chataway, J. (2005). The challenges of policy integration from an international perspective in Lyall, C. & Tait, J., Eds. New modes of governance. Chippenham, Ashgate Publishing Ltd: 159-176.

Nemes, G. (2004a) Constructing rural development: Models, institutions, policies and the Eastern enlargement PhD, Centre for the Rural Economy,  University of Newcastle,  Newcastle.

Nemes, G. (2004b). Integrated Rural Development, and the Reflexive Agency - Theory and practice in a Hungarian pilot programme. XI World Congress of Rural Sociology, Trondheim, Norway.

Nemes, G., High, C. & Huzair, F. (accepted). Reflexive agency and multi-level governance: mediating integrated rural development in South Transdanubia in Cheshire, L., Higgins, V. & Lawrence, G., Eds. International Perspectives on Rural Governance: New Power Relations in Rural Economies and Societies.

OECD (2001). Multifunctionality: Towards and analytical framework. OECD booklet, Organisation for Economic Co-operation and Development, Paris.

OESP (1997). Who are the question-makers? A participatory evaluation handbook,  Office of Evaluation and Strategic Planning, UNDP http://www.undp.org/eo/documents/who.htm,  Accessed:  2005.

Patton, M. (1990). Qualitative Evaluation Methods.

Pelling, M. & High, C. (2005). Understanding adaptation: what can social capital offer assessments of adaptive capacity? in Global Environmental Change A On-line postprint.

Pretty, J., Guijt, I., Thompson, J. & Scoones, I. (1995). A trainer's guide for participatory learning and action. London, IIED.

Ray, C. (1999a). Endogenous Development in the Era of Reflexive Modernity in Journal of Rural Studies 15(3). 257-267.

Ray, C. (1999b). Towards a Meta-Framework of Endogenous Development: Repertoires, Paths, Democracy and Rights in Sociologia ruralis 39(4). 522-537.

Rengasamy, S., Devavaram, J., Prasad, R., Erskine, A., Balamurugan, P. & High, C. (2001). Thaan Vuzha Nilam Tharisu: The Land Without a Farmer Becomes Barren - Policies that Work for Sustainable Agriculture and Rural Livelihoods in Virudhunagar District, Tamilnadu. London, IIED.

Richards, D. & Smith, M.J. (2002). Governance and public policy in the UK. Oxford.

Sanderson, I. (2002). Evaluation, policy learning and evidence-based policy making in Public administration 80(1). 1-22.

Saraceno, E. (1999). The evaluation of local policy making in Europe in Evaluation 5(4). 439-457.

Scheurich, J.J. (1997). Social relativism: (not quite) a postmodernist epistemology in Scheurich, J. J., Ed. Research method in the postmodern. London, The Falmer Press: 189.

Shannon, C.E. & Weaver, W. (1949). The mathematical theory of communication. Urbana, University of Illinois Press.

Shaw, P. (1997). Intervening in the shadow systems of organizations: Consulting from a complexity perspective in Journal of Organizational Change 10(3). 235-250.

Stoker, G. (1998). Governance as theory: five propositions in International Social Science Journal 50(155). 17-28.

Van der Ploeg, J.D., Renting, H., Brunori, G., Knickel, K., Mannion, J., Marsden, T., Roest, K., Sevilla-Guzman & E. Ventura, F. (2000). Rural development: from practices and policies towards theory in Sociologia Ruralis 40(4). 391-408.

Vorley, B. (2002). Sustaining Agriculture: Policy, Governance, and the Future of Family-based Farming. London, IIED.

Wadsworth, Y. (2001). Becoming responsive - and some consequences for evaluation as dialogue across distance in New Directions for Evaluation 2001(92). 45-58.

Webber, L.M. (1994). Participatory rural appraisal design: Conceptual and process issues in Agricultural systems 47. 107 - 131.

Wenger, E. (1999). Communities of practice: Learning, meaning and identity. Cambridge, Cambridge University Press.

White, S., Ed. (2003). Participatory video: Images that transform and empower. London, Sage.

Williams, P. (2002). The competent boundary spanner in Public Administration 80(1). 103-125.

Wilson, G.A. (2001). From productivism to post-productivism and back again?  Exploring the (un)changed natural and mental landscapes of European agriculture in Transactions of the Institute of British Geographers 26(1). 77-102.