President and CEO's Report for July 2013
This report continues my review of elements of the new Results Reporting dimension to our evaluation system (all three dimensions combined are called CN 3.0) that I began earlier this year. This month I will focus on the second element of this new dimension, which is called Results Logic and Measures. This second element of Results Reporting looks for a set of clear statements indicating how and why the charity’s activities are intended to lead to pre-defined outputs and outcomes.
To satisfy the requirements of this element, four yes/no questions are asked. They are:
Is the organization’s causal logic plausible?
Does the organization’s statement on how their activities lead to pre-defined outputs and outcomes make sense? This first question asks whether a reasonable person would conclude that a charity’s logic statement is plausible. If so, then that causal logic passes. By plausible we mean possible or likely under normal circumstances.
Let’s take the example of a job opportunities program. The logic statement for this hypothetical program is, “Our job classes provide our participants with the skills that they need to secure and hold a job.” On the face of it, this is clearly plausible. On the other hand, if the logic statement were, “Our lucky charm bracelets provide our participants with all that they need to secure and hold a job,” then the test would not be met.
Is there some indication of how much of the action is required to produce what effects?
The second question’s goal is to confirm that the charity has thought about how much of an activity or service is needed to produce their desired and intended effects, outputs and outcomes. In the medical community, this would be referred to as dosage. How much? How often? In increasing/decreasing amounts?
To return to the job employment opportunities example, the statement (“Our job classes provide our participants with the skills that they need to secure and hold a job.”) does tell us the result (secure and hold a job), but does not tell us how many or how long classes are required. As stated, this statement is insufficient to pass this test. In this case we look further to see if indications of quantity of classes are introduced elsewhere. For example, the organization might note that, “Completing four course modules over six months, and successfully demonstrating mastery of the material, is the required minimum participation to complete the basic program.” This statement would pass this test.
Is there some indication that the stated causal logic is based on reasonable evidence (e.g., references to other programs that have demonstrated results
following the same logic)?
It is one thing to make a statement and another thing to be able to strongly support the statement. The third question seeks to conclude whether the charity’s proposed actions have previously been shown to produce the intended results. The evidence could come from other organizations or from past experience from the charity itself.
In the hypothetical job opportunities program, the following statement would meet this test: “Over the past 5 years, 80 percent of our participants who pass four job courses within six months obtain employment within 6 weeks of graduating. For graduates of more than one year ago, 70 percent of these are still employed one year later.” Note that the evidence needs to speak to both aspects of the intended result – obtaining and retaining a job.
Are there specified measures (indicators) for outputs and outcomes to be collected and a plan to do so?
Question 4 seeks to confirm that organizations have a plan in place for data collection of specified indicators. It should be reasonable to assume that these indicators will capture the intended result. In other words, does the charity state what success looks like and how it will collect the data to know if it has been achieved?
For our hypothetical employment opportunities program we would expect to see something like: “Our graduates inform us when they obtain a job. We survey both our graduates and the employers twice a year to monitor their status. We maintain client employment histories for seven years.” This represents a data collection plan and a clear specified indicator (employment status).
In conclusion, Charity Navigator’s hope and goal is that we can help to highlight those charities that have a well functioning performance management system, from which the above questions can be easily answered. In addition, it is our hope that this will also spur on other charities to develop performance management systems so that they can become as effective as possible. As a result, charitable givers/social investors will have more highly effective charities to choose from and many more people and communities will be helped.
To learn more about the new Results Reporting dimension to our evaluation system, read our Concept Note.