1

Tutorial Overview

• TAKE HOME MESSAGES
• PURPOSE AND BENEFITS OF EVALUATION
• TYPES OF EVALUATION
• PROCESS EVALUATION
• OUTCOME EVALUATION
  • Outcomes: Short term, Intermediate, and Long term
• IMPACT EVALUATION
• OVERVIEW OF THE EVALUATION PROCESS
• SUMMARY
• QUICK CHECK
• REFERENCES

2

TAKE HOME MESSAGES

After completing this tutorial, you will be familiar with:

• The purpose and benefits of evaluation
• Process, outcome, and impact evaluation
• How to use each type of evaluation to assess your intervention

3

PURPOSE AND BENEFITS OF EVALUATION

Evaluation is the process of determining whether programs, interventions, or action plans are appropriate, adequate, effective, and efficient; and, if not, how to make them so.

Evaluation also shows if these strategies have unexpected benefits or create unexpected problems.

Reasons for Evaluation

Evaluation answers questions about your intervention that will ultimately help you:

• Assess the way the intervention is implemented
• Improve the intervention or some aspect of it
• Measure the intervention’s effectiveness
• Show efficient use of resources
• Demonstrate accountability

4

TYPES OF EVALUATION

There are 3 types of evaluation:

• Process evaluation
  • Used to measure aspects of how the intervention was implemented
  • Assesses intervention activities themselves, rather than their effect on the intended audience
  • Answers the question "Are my activities being carried out as planned?"
• Outcome evaluation
  • Used to measure the effect the intervention has on the intended audience
  • Assesses progress toward the intervention’s short term, intermediate, and long term goals
  • Answers the question "How did the intervention change the intended audience?"
• Impact evaluation
  • Used to measure the long term impact of the intervention
  • Assesses cause and effect relationships
  • Answers the question "What were the positive and negative consequences of the intervention?"

5

PROCESS EVALUATION

Process evaluations assess the delivery, or implementation, of your intervention.

• Answers the questions:
  • What happened and why?
  • How is that different from what was planned?

Process evaluation can:

• Assess the degree to which the intervention is being implemented as planned
• Link specific intervention strategies to outcomes
• Provide information for improving intervention elements
• Provide feedback for coalition members regarding the quality of implementation
• Provide information on accountability to people interested in your intervention such as funders, volunteers, or the target audience

6

OUTCOME EVALUATION

Outcome evaluation assesses the benefits or changes that occur in the intended audience as a result of the intervention.

• Answers the questions:
  • Did the intended audience change their [behavior, knowledge, attitudes, etc.] as a result of the intervention?
  • Did the intervention have the desired effect?

Outcome evaluations are important to making major decisions about intervention continuation, expansion, reduction, and funding.

7

OUTCOME EVALUATION cont'd

Outcomes: Short term, Intermediate, and Long term

Outcomes can be viewed as short term, intermediate, or longer term. These outcomes relate to different points in the life of the intervention.

The following table illustrates the differences between short term, intermediate and long term outcomes, followed by an example.

8

IMPACT EVALUATION

Impact evaluation measures the long term effects of the program – positive and negative; intended and unintended.

• Answers the question:
  • Overall, how did the intended audience benefit (or not) from the intervention?
  • What were the intended (and unintended) consequences of the intervention?
• An impact evaluation can be done by:
  • Comparing the same group of individuals before and after the intervention to assess changes
  • Comparing the individuals who received the intervention to other similar individuals who did not receive the intervention (within the community, or a similar neighboring community)

10

SUMMARY

Evaluation is a critical step in implementing an intervention.  Evaluation demonstrates exactly how your intervention works and how it affects the intended audience.  Results of the evaluation allow you to attribute changes in health outcomes to specific intervention activities, and identify any gaps and ways to improve your intervention.

In the next tutorial, Sustaining Coalition Action, you will learn how to sustain your coalition’s efforts in order to have a lasting impact in your community.

But first, a Quick Check…

16

REFERENCES

Adapted from:

• Dressler, L (2004) How to Achieve High-Commitment Decisions. Blue Wing Consulting. Boulder, CO.
• Rogow, F., Adelman, L., Poulain, R. and Cheng, J. (2008) UNNATURAL CAUSES Discussion Guide, a project of California Newsreel. ( www.unnaturalcauses.org)
• Nagy, J. and Fawcett, J. (2011).  Developing Successful Strategies: Planning to Win.  The Community Tool Box, Chapter 8, Section 4.  Available at http://ctb.ku.edu/en/tablecontents/sub_section_main_1088.aspx. 
• Task Force on Community Preventive Services.  The Guide to Community Preventive Services. New York: Oxford University Press; 2005. Available at http://www.thecommunityguide.org/index.html.  Accessed September 1, 2011.

Well done!

You have completed Tutorial 5: Designing an Evaluation Plan.  Click on Tutorial 6: Promoting and Sustaining Coalition Action on the left menu to go on to the next tutorial.

Want to learn more about program evaluation? Consider taking the online training course Fundamentals of Evaluation offer by the Diabetes Training and Technical Assistance Center (DTTAC).  Contact DTTAC for morning information at dttac@emory.edu .