Usefulness of summative evaluations to internal organizational planning

A recent article by the Stanford Social Innovation Review called “Drowning in Data”(http://www.ssireview.org/pdf/2006FA_feature_snibbe.pdf) quotes “For all their sophistication, they cannot prove that they are making an impact - a requirement that many funders now demand....“ While quoting this, the article is referring to outcome evaluation designs that don’t involve randomized control groups. Yes, it is true that there are severe threats to the validity of the results of an outcome evaluation that don’t involve randomized control groups.
But let’s for a moment forget how useful the results of these summative evaluations are, in proving to the funders about your impact. I feel that these evaluations are of great value to the organizations themselves. They provide a framework of results to the organizations with which they can ask larger and deeper questions.

I’m currently working with an organization which is trying to make a positive impact on youth in the Eastern Massachusetts area. They have a large client base in the order of tens of thousands. But they had never done a full fledged outcomes measurement studies to measure the possible impact that they were making. Because a funder asked for outcomes to be measured we piloted an implementation of an evaluation design that involved pre- and post-tests without having any control groups. I found that the results of these evaluations helped tremendously in looking at the big picture and to start asking questions within the organization. Agreed, the results are not entirely valid because there were no control groups. But as long as the organization is aware of this fact, following are some examples of questions that can be asked by looking at the results which will help them enormously in planning the future implementations of programs. ”Why are the outcome results of the “geographical area A” low compared to “geographical area B”? What differences exist in terms of the circumstances in which the youth operate in these two geographical areas? Can we connect these differences in the results back to the needs assessments that we had done earlier? Will having a focus group involving the youth of the “geographical area A” help us in any way to figure out better ways to tackle the issue?“

Another useful way to harness these evaluations to add value to non-profit organizations is to find ways to make them both summative and formative. It’s surprising how sometimes with little effort you can use an evaluation process originally designed for summative purposes to be made useful for formative purposes too.

The ultimate goal of every non-profit organization is to make sure that the issues related to their mission, for every segment that they are targeting, is getting resolved. It is important for them to prove to the funders that they as an organization are the ones making the impact and their model can be replicated cost-effectively. At the same time on a daily basis, that’s not what these non-profit organizations think about. What they think about is how they can best resolve the problems of all segments that they target given the limited resources that they have. Hence when you look at the results of an evaluation, you get to know how your resources can be re-organized to be put to better use. It doesn’t matter anymore whether it was the organization that was responsible for the better results in some segments. All that matters is that there are some segments, which are performing badly either compared to other segments that the organization is serving or compared to external benchmarks. How can the organizational resources be better tapped to perform better serve the under-performing segments? I feel that each organization should pro-actively look for ways to measure outcomes cost-effectively because it does help them to make better decisions related to their mission.