Evaluation Methods

Delivering Community Developments

Evaluating and Sustaining progress

Evaluation Methods

Evaluation Methods

Introduction

Ideally, evaluation is an on-going process. A good evaluation has its start at the beginning of the program or project. Some form of needs assessment should be conducted at the start of the planning process, to answer questions such as "What is the need (or problem or issue) that we are going to address?" and "How will the planned program/ activity help to address this need?". This will provide important points of reference for your evaluation.
While the program is in progress, you might want to conduct a "process" evaluation, which looks at how the project is running, what resources it is consuming, whether it is reaching the people you intended to, and whether the activities are co-ordinated and managed effectively.
An "outcome" evaluation looks at the results achieved by the project, both in terms of "outputs"; i.e. quantifiable data (e.g. number of trees planted, number of homeless people housed), and the measurable results or impacts achieved (e.g. the trees prevented further soil erosion; the health status of individuals who are homeless has been increased).
An economic evaluation looks at the costs of the project in comparison with the benefits of the program, which are also quantified into dollar values, if possible.
The following screens provide a brief overview of some of the more common evaluation methods used by community groups. Further reference materials are listed in the reference pdf.


Evaluation Methods

Qualitative or Quantitative?

There is often a debate about whether community projects and programs are best evaluated using quantitative methods or quantitative. Both are valuable sources of information.

Evaluation Methods
Evaluation Methods

Quantitative

Quantitative methods focus on things that can be counted. Often the "outputs" of the program will be counted; e.g., the number of participants involved, or number of sessions conducted. Outcomes may also be quantified by measuring the degree of change that has occurred. For example, participants might be given a pre-test before they start the program to measure their knowledge, skills or attitudes, then be given a post-test after they have completed the program. Comparing the group's average score for the two tests will give a measure of the change that has occurred within the group.
Quantitative methods use standardized measures with pre-determined response categories and usually involve statistical analysis. They can be applied to large groups and the data can be summarized and presented concisely. Also, they can be repeated with other groups for comparison and are often generalisable to other similar populations.

Qualitative Methods
· Focus groups · Diaries
· In-depth interviews · Consensus building
· Open-ended survey questions · Forums and discussion groups

Evaluation Methods

Qualitative

Qualitative methods are generally used to provide more in-depth information about individual experiences, perspectives and context. For example, you might ask participants to give a subjective report of the benefits they feel they have gained, or any change they have had in their attitudes or perceptions.
Qualitative methods include unstructured and open-ended questions, without pre-determined response categories. Because of the volume of information collected and the complexity of the analysis required, usually only small groups are involved. Another limitation of qualitative methods is that the findings of qualitative methods may not be generalisable to other groups

Quantitative Methods
· Surveys: drop-off, mail or telephone
· Tracking forms
· Service utilization statistics
· Direct measures of health indicators or behaviours (e.g. blood pressure, # of cigarettes smoked)

Evaluation Methods

Impact Evaluation

Impact assessments attempt to establish whether or not a program is producing the effects that were intended. They look at the magnitude of the results (i.e. whether there were large or small effects), whether there were other factors that might have been responsible for the outcomes and whether there were any other "unintended" results.
• Impact assessments identify the links between the various components of the program:
• Inputs money, staff, facilities
• Activities strategies, programs, services
• Outputs products of the activities, such as the number of services provided, number of classes taught, number of educational materials distributed
• Outcomes benefits or changes for individuals, or groups during or following participation (e.g., increased in literacy scores).
These components are often shown in a diagram called a project logic model.
A sample is provided in resources pdf

Evaluation Methods

Impact evaluations are used by program planners to identify the interventions that will best meet the needs of the program participants. They are also used by managers and administrators to document the value of a program to justify it to funders, and/or to help determine how to best allocate their resources. (Rossi et al. 1979: p. 161-165). Programs that have higher impacts are more likely to receive a larger share of the resources than are those with smaller impacts.
Most programs do not find it too difficult to measure inputs, activities and outputs, but often get stuck on measuring outcomes. Yet measuring outcomes is necessary if we are to be able to demonstrate that the program is actually making a difference. It can also help programs to:

• recruit and retain talented staff
• enlist and motivate able volunteers
• attract new participants
• engage collaborators
• garner support for innovative efforts
• win designation as a model or demonstration site
• retain or increase funding
• gain favourable public recognition
• focus staff effort on common goals and shared purposes
• communicate program results to stakeholders
• clarify the intended purpose of the program
• identify effective practices within the program
• successfully compete for resources/funding
• (see
Evaluation Methods

Participatory Evaluation

This type of evaluation uses the methods of participatory action research, which are based on the principles of adult education. "If one is concerned with increasing people's capacity to participate fully and gain some degree of control over their lives, then research methods themselves can be part of this method." (Hall, in Stinson, 1979:148)
To be truly effective, however, the community should be involved in the entire process, "…from the formulation of the problem to the discussion of how to seek solutions and the interpretation of findings." (Stinson, 1979:148)
There are many definitions of participatory research, but they all have the same basic themes:
• empowerment of participants
• collaboration through participation
• acquisition of knowledge
• social change.
The process that the researcher goes through to achieve these themes is a spiral of action research cycles consisting of four major phrases: planning, acting, observing and reflecting. (Zuber-Skerrit 1991:2, in www.scu.edu.au/schools/gcm/ar/arr/arow/rmasters.html#AR

Evaluation Methods

The basic idea behind participatory action research is that ordinary people, including program participants and/or clients, are normally shut out from the research process. By becoming involved, not only will they learn more about the program and its effectiveness, they will learn about research methods, how to obtain information and how to use it. The involvement of participants may also increase the appropriateness of the methods used, the degree of co-operation in the evaluation process and, ultimately, the quality of information obtained.
There is a strong emphasis on social change in participatory action research. Embedded in its practice is the goal of engaging in action that "reverses inequalities, empowers the have nots, and ultimately transforms society so power is based in grassroots organizations and individuals." (Community Development Society www.comm-dev.org/par-is.htm, June 2002)


Evaluation Methods

Empowerment evaluation, which is closely related to participatory action research, aims to "use evaluation concepts, techniques and findings to foster improvement and self-determination. It recognizes that in order to reflect an on-going change process, the evaluation process also needs to be on-going. Program participants establish their own goals and assess their progress using both qualitative measures (e.g.; reflection on their experience) and quantitative measures (e.g.; self-evaluation procedures). In a continuous cycle, they plan, evaluation, then re-shape their plans and strategies based on the results of the evaluation. The role of the researcher is empowerment evaluation is that of coach and/or facilitator.
Steps in empowerment evaluation generally include:
• taking stock of where the program stands now, including assessing its strengths and weaknesses
• establishing goals to determine where the participants want the program to go in the future, with an emphasis on program improvement
• developing strategies to accomplish program goals and objectives
• determining the type of evidence required to document their progress and how it will be obtained (Chelimsky and Shadish, 1997; pp. 382-384)

Empowerment evaluation

Evaluation Methods

Economic Evaluations

Economic evaluations are mostly concerned with the question "Is it worth it?". They are important because resources are usually scarce, and often there are hard decisions that need to be made as where those resources are best placed. There are many different ways of looking at the costs of a program and relating them to the effectiveness of the program:
•
• Cost Descriptions itemizes the costs involved in delivering the program
• Cost-Outcome Descriptions identifies how much it costs to achieve a particular outcome
• Cost-Effectiveness Analysis compares the costs of two or more alternative methods which achieve a particular outcome with varying degrees of success
• Cost-Benefit Analysis compares two or more alternatives whose costs and consequences are expressed in terms of the dollar value
• Cost-Utility Analysis compare the costs of two or more alternatives whose outcomes are judged by their perceived usefulness

Evaluation Methods

In order to conduct an economic evaluation, you must be able to identify the costs involved in the program you want to evaluate. Depending on the nature of the program, this may require considerable time and specialized skills. In most Healthy Community projects it is very difficult to put a dollar figure to the benefits that are attained but, if it is possible, it is highly recommended. Tangible economic benefits will help ensure that your project is well-received.

Evaluation Methods

Ethics in Evaluation

Economic evaluations are mostly concerned with the question "Is it worth it?". They are important because resources are usually scarce, and often there are hard decisions that need to be made as where those resources are best placed. There are many different ways of looking at the costs of a program and relating them to the effectiveness of the program:
• Cost Descriptions itemizes the costs involved in delivering the program
• Cost-Outcome Descriptions identifies how much it costs to achieve a particular outcome
• Cost-Effectiveness Analysis compares the costs of two or more alternative methods which achieve a particular outcome with varying degrees of success
• Cost-Benefit Analysis compares two or more alternatives whose costs and consequences are expressed in terms of the dollar value
• Cost-Utility Analysis compare the costs of two or more alternatives whose outcomes are judged by their perceived usefulness

Evaluation Methods

If it is, proper care must be taken to ensure that the confidentiality is securely maintained (e.g. use codes, not names on questionnaires). If not, participants should be informed how and to whom the information will be communicated.
• Protection of the people being studied participation should be voluntary, with full disclosure as to the purpose of the evaluation, and if there is any aspect that causes discomfort to a participant they must be allowed to withdraw from the study.
•
• Possibility of negative side effects that may affect the program and/or participants
• Different needs and interests of various stakeholders; for example, program staff and volunteers may feel threatened by an evaluation process. It might be helpful to use a framework of quality improvement as it may be seen as less threatening.

Delivering Community Developments

End of Unit:
Evaluating and Sustaining Progress - Evaluation Methods
Click next below to advance to the next topic