Participatory Evaluation I
Participatory Evaluation I
Original Article
doi:10.1111/j.1365-2214.2006.00657.x
Abstract
Background There is a need to study methodologies for evaluating social development projects. Traditional methods of evaluation are often not able to capture or measure the spirit of change in people, which is the very essence of human development. Using participatory methodologies is a positive way to ensure that evaluations encourage an understanding of the value of critical analysis among service providers and other stakeholders. Participatory evaluation provides a systematic process of learning through experiences. Methods Practical experiences of conducting a number of evaluation studies in social development projects have led the author to develop four basic principles of participatory evaluation strategies. This has been further conceptualized through an extensive literature search. The article develops and shares these principles through descriptions of eld experiences in Asia. Results The article illustrates that the role of any evaluation remains a learning process, one which promotes a climate of reection and self-assessment. It shows how using participatory methods can create this environment of learning. However, one needs to keep in mind that participatory evaluation takes time, and that the role and calibre of the facilitator are crucial. Conclusion Participatory evaluation methods have been recommended for social development projects to ensure that stakeholders remain in control of their own lives and decisions.
Keywords facilitator, eld experience, participatory evaluation, stakeholder Correspondence: Brinda Crishna, 3 B Dover Palace, 6, Dover Road, Kolkata 700016, West Bengal, India E-mail: [email protected]
Original Article
Introduction
This article claries the principles of participatory methods for the evaluation of community development projects for disadvantaged groups through examples of eld experiences in India, Bangladesh and Nepal. It denes four basic principles of participatory evaluation strategies drawn from practical experiences in the eld and further conceptualized through an extensive literature search. It analyses the appropriateness of using participatory methods for evaluating community-based development projects. In the past two decades there has been a great change in thinking about development programmes for people who have been marginalized, the types of services best suited to full their basic human rights and the role they will play themselves in fullling these needs. No longer are programmes accepted as
mere conduits for service delivery, but they are seen as social mobilizers and a channel for empowering people. Concepts of participatory development and research have been born out of such beliefs the capacity of marginalized populations to assess and solve their own development problems (Garaway 1995; Chambers 1997a; Vlaenderen 2001). As a natural progression of project implementation, there is the need for evaluations. This has been a prerequisite for most funding agencies in developing countries, and project implementers are increasingly accepting the value of evaluation as an integral part of project development.
217
218
B. Crishna
process for learning through experiences. It provides a means to look constructively at the strengths and weaknesses in projects, and use the lessons learned to improve planning and implementation measures. It should measure the distance travelled (Dewson et al. 2000). It should be a partnership between everyone involved, not be seen as an inspection or a test. It should not turn people into passive objects of knowledge, but rather empower those traditionally objectied to become the knowers, and be in control of the decisions being made about their lives (Nelson & Wright 1997). That is, the approach embodies formative evaluation, rather than summative evaluation where externally developed measures are applied and objectivity is the key principle (McConachie 1999). However, when it comes to actual practice, most non-government organizations (NGOs) and government organizations in the developing world have been hesitant to carry out evaluations. There are very real reasons for this hesitancy. Experience has shown that programme planners and implementers often have a very limited understanding of what evaluation means and are suspicious of it. Negative feelings have developed due to years of lling in survey forms for government departments. There is no value put to this activity as it results in few or no change. Evaluations have been conducted primarily by external professional evaluators, leading to service providers feeling insecure under scrutiny. The smaller the NGO, the greater their sense of insecurity is. They are often anxious that their programmes are being checked up on, and that further funding depends on success, resulting in NGOs shying away from acknowledging their problems. Traditional methods of evaluation are often not able to capture or measure the spirit of change in people, which is the very essence of human development. Using participatory methodologies is a positive way to ensure that evaluations encourage an understanding of the value of critical analysis among service providers and other stakeholders. Vlaenderen (2001, p. 343) stresses that there is a dialectical relationship between empowerment and participation.
MODELS denition of models: Models include stakeholderbased, empowerment evaluation (Mathie & Greene 1997; Schnoes et al. 2000), participatory action research, participatory rural appraisal (Chambers 1997b) and realistic evaluation (Pawson & Tilley 1997). This is by no means a comprehensive list, but many of the features, objectives and activities are common among all the models. PARTICIPATION what participation means: it involves understanding the importance of dialogue and partnership (Kelly & Vlaenderen 1995), issues around levels, extent and complexities of participation (Chambers 1997a; Naylor et al. 2002). STAKEHOLDERS the understanding of who stakeholders are or should be: these stress the importance of including stakeholders and the problems of doing so at a practical level (Mathie & Greene 1997; Townsend 1997; Gregory 2000). POWER the role of power: these articles examine negative inuences and the creation of problems in the participatory process (Maher 1997; Mathie & Greene 1997; Gregory 2000). FACILITATOR an understanding of the role of the facilitator: articles highlight how crucial it is, and the need for personal awareness (Garaway 1995; Townsend 1997; Schones et al. 2000).
1. Everyone involved in the programme shares control over the process of evaluation
Who are the stakeholders and who chooses them? Often those who are most accessible to evaluators are those who have more
2006 The Author Journal compilation 2006 Blackwell Publishing Ltd, Child: care, health and development, 33, 3, 217223
Participatory evaluation
219
power, who have more control, and who are more outspoken, so it could mean that a community in India is represented by the sarpanj (the village head), the Brahmin (the priest), the local political leader accompanied by a few quiet villagers who stay in the background and agree to everything, or the male heads of families who make life-changing decisions without consulting the women in their families. For participation to take place in its truest sense, evaluation facilitators need to ensure that stakeholders represent all those who are involved and will be affected by the project. Mathie and Greene (1997) go a step further to say that diversity among the stakeholder group also needs to be maintained. To achieve this, it requires an inherent belief within facilitators that beneciaries of projects (whether they are poor people, the aged, children surviving on the streets, people with disabilities or without literacy, or women in orthodox rural villages or any other such marginalized groups) have the ability to analyse their own lives and make decisions about them (Crishna 1999).
The strategy used 1 A series of face-to-face meetings was arranged with the company representatives in the eld, local village representatives chosen by the villagers themselves, and the international bank, with the objective of setting up a transparent process of consultation, encouraging actual dialogue, and sharing concerns and ideas on an equal platform. A timetable for continued ongoing meetings was set up. In many cases, womens meetings were held separately. 2 A livelihood assessment, with the help of local development NGOs and villagers, with the aim of putting together a choice of alternative income generation activities which the communities felt would be appropriate for them. 3 Short on going training sessions on understanding how the compensation of getting money for land would work for them, the complexities of using banks as a way of saving their income, how the Trust Fund would work and other such need-based issues. Results When the work on the mine nally started, there was active co-operation from the villagers. The majority of the villagers saw this as a process of improved development, a change from the old to the new.
2. The objectives are set in a group and jointly with all the people concerned in the programme, keeping in mind that every one has his or her own agenda
The objectives are set in a group and have to meet the needs of all the players: those who are providing the funding, those who are receiving the funding and are directly responsible for the running of the programme, and the beneciaries of the programme.
2006 The Author Journal compilation 2006 Blackwell Publishing Ltd, Child: care, health and development, 33, 3, 217223
220
B. Crishna
Everyone had his or her own objectives and agendas as to why they were participating in the study:
The strategy used Initial planning meetings and an objectivesetting workshop were held with the funding agency, the Dhaka agency, and their partners and members from the community. Everyone was encouraged to share his or her ideas, and consensus was obtained. The objectives were then listed, and participants realized that the nal list actually met the needs of everyone, thus allowing the study to be initiated on a positive note. Results
with disabilities. The CBR programme had been running for 5 years before it was able to get international support. The programme is an impressive one as it has grown slowly and has been dened by the needs of the community, is run by members from the community, and works in tandem with general development programmes. It uses local resources and builds on existing strengths within its community. I rst visited it when I was asked to conduct a skill-training workshop for their CBR workers. I was deeply impressed by how well the organization had understood the concepts and philosophy of CBR and seemed to have internalized it into their lives. Three years later I was asked by the NGO to come and conduct a mid-term review of the CBR project, as the funders needed this information to decide on further funding and continued assistance. The NGO had managed to convince the funder that they would like to choose their own evaluator. During my initial meetings, I was startled by the animosity and lack of trust that the NGO showed in having to comply with this request. I learned that they had undergone a review 1 year after they had received funding and had been deeply disillusioned with the methods that had been used and the conclusions drawn by the evaluators. The evaluators had been two foreign specialists in disability, who had taken 10 days to conduct the evaluation. They had used questionnaires to gather data and had carried out the entire exercise without any consultations with the NGO themselves. One month after this evaluation was completed, the NGO was asked to a meeting and was given the results of the evaluation. The results were negative and critical, and had left the NGO feeling insecure and angry. As a result of the evaluation, the funders put in many more systems for monitoring into the programme, resulting in more paperwork and less time for programme work. The strategy used
3. Working out the difculties faced by everyone helps in strengthening the programme
All the stakeholders need to be able to visualize the change. If people are incapable of seeing the developmental process going on, or how the lessons being learned are encouraging change, it remains the role of the facilitator to create an environment of sharing and learning for this realization to take place.
2006 The Author Journal compilation 2006 Blackwell Publishing Ltd, Child: care, health and development, 33, 3, 217223
Participatory evaluation
221
ers, analysing results, drawing conclusions about strengths and constraints, making recommendations, and drafting the nal report, in other words, how participatory research works; to facilitate the actual evaluation of the CBR programme using all that they were learning. At the end, 1 day was spent in analysing the 3-month-long process and the participatory methodology used. Results
and that children would be able to judge progress in the programme. The strategy used
2006 The Author Journal compilation 2006 Blackwell Publishing Ltd, Child: care, health and development, 33, 3, 217223
222
B. Crishna
the attitude and values projected that transforms the evaluation from just another exercise in nding out whether the project works or not, to an empowering experience for all concerned. The attitude has to be one of understanding and reection, having the capacity of look within and continuously challenging ones own professional ideologies and understanding of social truths, of development processes, of how the poorest of the poor think and understand their own lives, of moving away from clichs and generalities about marginalized people and listening to what is. This remains the most crucial part of the process, and one that makes or breaks its value.
which aspect of development, the role of any evaluation remains a learning process, one that promotes a climate of reection and self-assessment. The role of the facilitator is crucial at every stage of the process. In the four examples given to illustrate the four principles of participatory evaluation, the facilitator has had a different role in each: he or she has been a broker with the power to involve all the stakeholders, an artist who helps create a vision, a mediator who facilitates understanding, and a counsellor who helps build self-esteem and worth.
Acknowledgements
This article is the result of a 3-month professional fellowship which was awarded to the author from the Child Care, Health and Development Trust. The author would like to thank Child Care, Health and Development for giving her the opportunity to reect upon and study the intricacies of participatory evaluation methodologies and thus add academic value to her experiences in the eld. She would like to thank Dr Helen McConachie and Dr Pauline Pearson for their guidance and contribution to her learning.
References
Chambers, R. (1997a) Whose Reality Counts? Putting the First Last. Intermediate Technology Publications, London, UK. Chambers, R. (1997b) Paradigm shifts and the practice of participatory research and development. In: Power and Participatory Development, Theory and Practice (eds N. Nelson & S. Wright), pp. 3042. Intermediate Technology Publications, London, UK. Crishna, B. (1999) What is community-based rehabilitation a view from experience. Child: Care, Health and Development, 25, 2735. Dewson, S., Eccles, J., Tackey, N. D. & Jackson, A. (2000) Guide to Measuring Soft Outcomes and Distance Travelled. Institute for Employment Studies, Brighton, UK. Garaway, G. B. (1995) Participatory evaluation. Studies in Educational Evaluation, 21, 85102. Gregory, A. (2000) Problamatizing participation a critical review of approaches to participation in evaluation theory. Evaluation, 6, 179 199. Kelly, K. & Vlaenderen, H. V. (1995) Evaluating participation processes in community development. Evaluation and Program Planning, 18, 371383. McConachie, H. (1999) Conceptual frameworks in evaluation of multidisciplinary services for children with disabilities. Child: Care, Health and Development, 25, 101133. Maher, V. (1997) Participatory research on non-European immigration to Italy. In: Power and Participatory Development, Theory and Practice (eds N. Nelson & S. Wright), pp. 105113. Intermediate Technology Publications, London, UK.
Conclusion
Experiences of work have been drawn from different countries and in varied elds of community developmental work. The article thus illustrates that no matter which country or
2006 The Author Journal compilation 2006 Blackwell Publishing Ltd, Child: care, health and development, 33, 3, 217223
Participatory evaluation
223
Mathie, A. & Greene, J. C. (1997) Stakeholder participation in evaluation: how important is diversity? Evaluation and Program Planning, 20, 279285. Naylor, P. J., Wharf-Higgens, J., Blair, J., Green, L. & OConnor, B. (2002) Evaluating the participatory process in a community-based heart health project. Social Science and Medicine, 55, 11731187. Nelson, N. & Wright, S. (1997) Participation and power. In: Power and Participatory Development, Theory and Practice (eds N. Nelson & S. Wright), pp. 118. Intermediate Technology Publications, London, UK. Pawson, R. & Tilley, N. (1997) Realistic Evaluation. Sage Publications, London, UK.
Schnoes, C. J., Murphy Berman, V. & Chambers, J. M. (2000) Empowerment evaluation applied: experiences, analysis, and recommendations from a case study. American Journal of Evaluation, 21, 5364. Townsend. J. (1997) Who speaks for whom? Outsiders re-present women pioneers of the forests of Mexico. In: Power and Participatory Development, Theory and Practice (eds N. Nelson & S. Wright), pp. 95104. Intermediate Technology Publications, London, UK. Vlaenderen, H. V. (2001) Evaluating development programs: building joint activity. Evaluation and Program Planning, 24, 343352.
2006 The Author Journal compilation 2006 Blackwell Publishing Ltd, Child: care, health and development, 33, 3, 217223