Whether your evaluation includes formal or informal research procedures, you’ll still have to collect and analyze data, and there are some basic steps you can take to do so.

1.  Implement your measurement system.  In Section 3 of this chapter, we discussed designing an observational system to gather information.  Now it’s time to put that system in place. 

·         Clearly define and describe what measurements or observations are needed.  The definition and description should be clear enough to enable observers to agree on what they’re observing and reliably record data in the same way. 

·         Select and train observers.  Particularly if this is part of a participatory process, observers need training to know what to record; to recognize key behaviors, events, and conditions; and to reach an acceptable level of inter-rater reliability (agreement among observers).

·         Conduct observations at the appropriate times for the appropriate period of time.  This may include reviewing archival material; conducting interviews, surveys, or focus groups; engaging in direct observation; etc.

·         Record data in the agreed-upon ways.  These may include pencil and paper, computer (using a laptop or handheld device in the field, entering numbers into a program, etc.), audio or video, journals, etc.

2.  Organize the data you’ve collected.  How you do this depends on what you’re planning to do with it, and on what you’re interested in.

·         Enter any necessary data into the computer.  This may mean simply typing comments, descriptions, etc., into a word processing program, or entering various kinds of information (possibly including audio and video) into a database, spreadsheet, a GIS (Geographic Information Systems –

·         Transcribe any audio- or videotapes.  This makes them easier to work with and copy, and allows the opportunity to clarify any hard-to-understand passages of speech.

·         Score any tests and record the scores appropriately.

·         Sort your information in ways appropriate to your interest.  This may include sorting by category of observation, by event, by place, by individual, by group, by the time of observation, or by a combination or some other standard.

·         When possible, necessary, and appropriate, transform qualitative into quantitative data.  This might involve, for example, counting the number of times specific issues were mentioned in interviews, or how often certain behaviors were observed.

3.  Conduct data graphing, visual inspection, statistical analysis, or other operations on the data as appropriate.  We’ve referred several times to statistical procedures that you can apply to quantitative data.  If you have the right numbers, you can find out a great deal about whether your program is causing or contributing to change and improvement, what that change is, whether there are any expected or unexpected connections among variables, how your group compares to another you’re measuring, etc.  There are other excellent possibilities for analysis besides statistical procedures, however.  A few include:

·         Simple counting, graphing and visual inspection of frequency or rates of behavior, events, etc., over time.

·         Using visual inspection of patterns over time to identify discontinuities (marked increases, decreases) in the measures over time (sessions, weeks, months).

·         Calculating the mean (average), median (midpoint), and/or mode (most frequent) of a series of measurements or observations.  What was the average blood pressure, for instance, of people who exercised 30 minutes a day at least five days a week, as opposed to that of people who exercised two days a week or less?

·         Using qualitative interviews, conversations, and participant observation to observe (and track changes in) the people or situation.  Journals can be particularly revealing in this area because they record people’s experiences and reflections over time.

·         Finding patterns in qualitative data.  If many people refer to similar problems or barriers, these may be important in understanding the issue, determining what works or doesn’t work and why, or more.

·         Comparing actual results to previously determined goals or benchmarks.  One measure of success might be meeting a goal for planning or program implementation, for example. 

4.  Take note of any significant or interesting results.  Depending on the nature of your research, results may be statistically significant (the 95% or better certainty that we discussed earlier), or simply important or unusual.  They may or may not be socially significant (i.e., large enough to solve the problem). There are a number of different kinds of results you might be looking for.

·         Differences within people or groups. If you have repeated measurements for individuals/groups over time, we can see if there are marked increases/decreases in the (frequency, rate) of behavior (events, etc.) following introduction of the program or intervention. When the effects are seen when and only when the intervention is introduced – and if the intervention is staggered (delayed) across people or groups – this increases our confidence that the intervention, and not something else, is producing the observed effects.

·         Differences between or among two or more groups.  If you have one or more randomized control groups in a formal study (groups that are drawn at random from the same population as the group in your program, but are not getting the same program or intervention, or are getting none at all), then the statistical significance of differences between or among the groups should tell you whether your program has any more influence on the dependent variable(s) than what’s experienced by the other groups. 

·         Results that show statistically significant changes.  With or without a control or comparison group, many statistical procedures can tell you whether changes in dependent variables are truly significant (or not likely due to chance). These results may say nothing about the causes of the change (or they may, depending on how you’ve structured your evaluation), but they do tell you what’s happening, and give you a place to start.  

·         Correlations.  Correlation means that there are connections between or among two or more variables.  Correlations can sometimes point to important relationships you might not have predicted.  Sometimes they can shed light on the issue itself, and sometimes on the effects of a group’s cultural practices. In some cases, they can highlight potential causes of an issue or condition, and thus pave the way for future interventions.

Correlation between variables doesn’t tell you that one necessarily causes the other, but simply that changes in one have a relationship to changes in the other.  Among American teenagers, for instance, there is probably a fairly high correlation between an increase in body size and an understanding of algebra.  This is not because one causes the other, but rather the result of the fact that American schools tend to begin teaching algebra in the seventh, eighth, or ninth grades, a time when many 12-, 13-, and 14-year-olds are naturally experiencing a growth spurt. On the other hand, correlations can reveal important connections.  A very high correlation between, for instance, the use of a particular medication and the onset of depression might lead to the withdrawal of that medication, or at least a study of its side effects, and increased awareness and caution among doctors who prescribe it.  A very high correlation between gang membership and having a parent with a substance abuse problem may not reveal a direct cause-and-effect relationship, but may tell you something important about who is more at risk for substance abuse.

·         Patterns.  In both quantitative and qualitative information, patterns often emerge: certain health conditions seem to cluster in particular geographical areas; people from a particular group behave in similar ways; etc.  These patterns may not be specifically what you were looking for or expected to find, but they may either be important in themselves or shed light on the areas you’re interested in.  In some cases, you may need to subject them to statistical procedures (regression analysis, for example) to see if, in fact, they’re random, or if they constitute actual patterns.

·         Obvious important findings.  Whether as a result of statistical analysis, or of examination of your data and application of logic, some findings may stand out.  If 70% of a group of overweight participants in a healthy eating and physical activity program lowered their weight and blood pressure significantly, compared to only 20% of a similar group not in the program, you can probably assume that program may have been effective.  If there’s no change whatsoever in education outcomes after two years of your education program, then you’re either running an ineffective program, or you’re simply not reaching those who are most likely to have poorer outcomes (which can also be interpreted to mean you’re running an ineffective program.)

Not all important findings will necessarily tell you whether your program worked, or what is the most effective method.  It might be obvious from your data collection, for instance, that, while violence or roadway injuries may not be seen as a problem citywide, they are much higher in one or more particular areas, or that the rates of diabetes are markedly higher for particular groups or those living in areas with greater disparities of income. If you have the resources, it’s wise to look at the results of your research in a number of different ways, both to find out how to improve your program, and to learn what else you might do to affect the issue.

5.  Interpret the results.  Once you’ve organized your results and run them through whatever statistical or other analysis you’ve planned for, it’s time to figure out what they mean for your evaluation.  Probably the most common question that evaluation research is directed toward is whether the program being evaluated works or makes a difference.  In research terms, that often translates to “What were the effects of the independent variable (the program, intervention, etc.) on the dependent variable(s) (the behavior, conditions, or other factors it was meant to change)?”   There are a number of possible answers to this question:

·         Your program had exactly the effects on the dependent variable(s) you expected and hoped it would.  Statistics or other analysis showed clear positive effects at a high level of significance for the people in your program and – if you used a multiple-group design – none, or far fewer, of the same effects for a similar control group and/or for a group that received a different intervention with the same purpose.  Your early childhood education program, for instance, greatly increased development outcomes for children in the community, and also contributed to an increase in the percentage of children succeeding in school.

·         Your program had no effect.  Your program produced no significant results on the dependent variable, whether alone or compared to other groups.  This would mean no change as a result of your program or intervention.

·         Your program had a negative effect.  For instance, intimate partner violence increased (or at least appeared to) as a result of your intervention. (It is relatively common for reported events, such as violence or injury, to increase when the intervention results in improved surveillance and ease of reporting).  

·         Your program had the effects you hoped for and other effects as well. 

·         These effects might be positive.  Your youth violence prevention program, for instance, might have resulted in greatly reduced violence among teens, and might also have resulted in significantly improved academic performance for the kids involved.

·         These effects might be neutral.  The same youth violence prevention program might somehow result in youth watching TV more often after school.

·         These effects might be negative.  (These effects are usually called unintended consequences.) Youth violence might decrease significantly, but the incidence of teen pregnancies or alcohol consumption among youth in the program might increase significantly at the same time.

·         These effects might be multiple, or mixed. For instance, a program to reduce HIV/AIDS might lower rates of unprotected sex but might also increase conflict and instances of partner violence. Your program had no effect or a negative effect and other effects as well.  As with programs with positive effects, these might be positive, neutral, or negative; single or multiple; or consistent or mixed.  

• If your analysis gives you a clear indication that what you’re doing is accomplishing your purposes, interpretation is relatively simple: You should keep doing it, while trying out ways to make it even more effective, or while aiming at other related issues as well. 

If your analysis shows that your program is ineffective or negative, however – or, for that matter, if a positive analysis leaves you wondering how to make your successful efforts still more successful – interpretation becomes more complex.  Are you using an absolutely wrong approach?  Are you using an approach that could be effective, but is poorly implement?  Is there a particular contributing factor you’re failing to take into account?  Are there barriers to success – of culture, experience, personal characteristics, systematic discrimination – present in the population from which participants are drawn?   Are there particular components or elements you can change to make your program more effective, or should you start again from scratch?  What should you address to make a good program better? Careful and insightful interpretation of your data may allow you to answer questions like these.  You may be able to use correlations, for instance, to generate hypotheses about your results.  As we’ve discussed, if positive or negative changes in particular variables are consistently associated with positive or negative changes in other variables, the two may be connected.  (The word “may” is important here.  The two may be connected, but they may not, or both may be related to a third variable that you’re not aware of or that you consider trivial.)  Such a connection can point the way toward a factor (e.g., access to support) that is causing the changes in both variables, and that must be addressed to make your program successful.  

Correlations may also indicate patterns in your data, or may lead to an unexpected way of looking at the issue you’re addressing. You can often use qualitative data to understand the meaning of an intervention, and people’s reactions to the results. The observation that participants are continually suffering from a variety of health problems may be traced, through qualitative data, to nutrition problems (due either to poverty or ignorance) or to lack of access to health services, or to cultural restrictions (some Muslim women may be unwilling – or unable because of family prohibition – to accept care and treatment from male doctors, for example). Once you have organized your data, both statistical results and anything that can’t be analyzed statistically need to be analyzed logically.  This may not give you convincing information but it will almost undoubtedly give you some ideas to follow up on, and some indications of connections and avenues you might not yet have considered.  It will also show you some additional results – people reacting differently than before to the program, for example.  The numbers can tell you whether there is change, but they can’t always tell you what causes it or why (although they sometimes can), or why some people benefit while others don’t.  Those are often matters for logical analysis, or