How Do We Plan To Evaluate Results Reporting?
“This is the most important work being done in the nonprofit sector.” ~ Paul Brest, Professor of Law, Emeritus and Former Dean of Stanford Law School and former president of the William and Flora Hewlett Foundation
“This is a really good start to a really tough problem.” ~ Craig Newmark, Founder, Craigslist
“These standards are exactly the right sort of step forward.” ~ Sean Stannard-Stockton, Director of Investments and Partner at Ensemble Capital Management
“CN 3.0 represents one of the most innovative approaches to advancing the nonprofit space that I have seen.” ~ Kerry Vaughan, Technology & Innovation Manager at the Laura and John Arnold Foundation
Mission-related results are the very reason that charities exist. Effective charities have a strong focus on results; on the outcomes and impact resulting from their work. This dimension evaluates how well charities report on their results.
What We’re Looking For
When we refer to a charity’s results, we are referring to what we expect them to achieve with our donations. In Results Reporting, we are specifically focusing on the way charities come to know, use and share their results with stakeholders including donors. In doing so, we hope to shift the paradigm of results reporting from selectively reporting, such as storytelling, that may not be representative of overall performance, to reporting on demonstrably important measures, and showing how the organization learns and improves based on those measures.
We believe effective charities manage their performance and thereby know and act on their results. These high performing organizations use early results as milestones to important longer-term results.
A Developmental Approach
We recognize that the majority of charities are in the early stages of measuring and reporting results, and do not yet have the necessary systems in place to meet the Results Reporting rating criteria. We will engage with charities with the intention of encouraging and incentivizing progressive improvement in results measurement and reporting practices over the coming years.
As charities demonstrate improved results reporting practices, the rating criteria will evolve to keep raising the bar of reporting requirements as more and more charities aspire to and become higher performing.
Currently this data is presented for informational purposes only. We do not intend to rate charities on this information until we have gathered the data for every organization in our database. We expect this process to be complete in 2016 provided we are able to secure the needed funding and related resources to accomplish this.
What Data We Use
We consider five rating elements when examining reporting results:
Element One: Alignment of Mission, Solicitations and Resources
We determine if the charity’s funding solicitation materials (on a donations page, public financial reports, or for the first year the charity is reviewed only - materials sent to CN) are in accordance with how it allocates its resources as per the program expenses break out on the most recent Form 990. All subsequent rating elements are evaluated for each individual program area until at least 60% of overall program expenses are represented.
In this early, developmental phase of evaluating charities results reporting, we have determined that the Element One (Alignment of Mission, Solicitation and Resources) metric is a more appropriate fit in the Accountability and Transparency dimension of our methodology. In light of this, we anticipate moving Element One into that section of our methodology prior to the implementation of rating charities on results reporting. As a result, Alignment of Mission, Solicitation and Resources will start impacting each organization's rating much sooner than results reporting metrics.
Element Two: Results Logic and Measures
Is the organization’s causal logic plausible?
Is the organization’s statement on how their activities lead to pre-defined outputs and outcomes likely under normal circumstances? (Outputs are the immediate results of the activities – for example, how many people were fed or sheltered, or how many roads built.Outcomes are the medium- and long term changes. They might be improvements in nutritional status or health indicators.)
Is there an indication of how much of the action is required to produce the pre-defined outputs and outcomes?
Has the charity given thought to the required amount and time that is needed to produce the pre-defined outputs and outcomes?
Is the causal logic based on reasonable evidence?
Does the report include reference to evidence that these actions have in fact been demonstrated to produce the intended results? References could be evidence from other organizations or from past activity by the charity itself.
Are there specified measures (indicators) to be collected and a plan to do so?
Does the charity state what success looks like and how it will collect the data to know if it has been achieved?
- Is the organization’s causal logic plausible?
Element Three: Validators
For the purposes of Results Reporting assessment, we recognize the work of other organizations which, through their processes, standards or analysis, examine or assess the charities we evaluate. Charity Navigator realizes that it is not the only organization that evaluates charities; and that the evaluation processes of many other organizations have merit. Our intent therefore is to give charities credit for making public on their own websites or other materials the results of evaluations or analyses from these other “validator” organizations.
Charities indicate which organizations they are members in good standing of, or show evidence of their participation in the organization on their website, or in materials emailed to Charity Navigator.
- Charities indicate which organizations they are members in good standing of, or show evidence of their participation in the organization on their website, or in materials emailed to Charity Navigator.
Element Four: Constituent Voice
This element assesses whether and how well a charity collects and publishes feedback from it primary constituents. (Primary constituents are the people who are meant to be the direct recipients of benefits created by the organization’s actions.)
Does the charity publish feedback data from its primary constituents?
Is there a rigorous method of data collection using the same questions and capturing the answers consistently?
Does this data indicate if it is representative of all primary constituents?
The feedback data must in some way indicate why the data set could be seen to be representative of the whole group.
Does this data include and explanation of why the organization believes the feedback is frank and honest?
Is there an explanation in the feedback data that explains why the feedback is frank and honest, for example, is it collected anonymously, and/or using multiple methods etc.
Is this data presented in a way that shows changes over time going back at least one year?
Does the data show how answers to the same questions to the same constituent group change over time?
Does this data include questions that speak to the organization’s effectiveness?
Do any of the questions address how well the organization’s activities meet the need of the respondents? Do any questions ask if the organization’s activities lead to improvements or positive changes in their lives?
Does the organization report back to its primary constituents what it heard from them?
Have the findings from the data collection process been reported back to those who provided the data, and others like them?
- Does the charity publish feedback data from its primary constituents?
Element Five: Published Evaluation Reports
This element assesses whether a charity is publishing regular evaluation reports that cover the results of a majority of their programs.
- Does the charity publish evaluation reports that cover the results of its programs at least every five years?
- Are those reports based on recognized techniques to understand their results?
- Does the charity explain what, if anything, it is changing as a result of the findings in the evaluation report?
Charity Navigator believes that a shift in paradigm is critically important to the future of the non profit world. We aim to recognize charities that do a better job of reporting their results. Here, we mean those that do not just put out promotional marketing materials, but that report on demonstrably important measures and show how the organization learns and improves based on those measures. As more charities report their results, givers and social investors will have more and better information to inform their decisions. These are the organizations that are worthy of our highest ratings and the maximum support from wise donors who want to invest in those charities that provide the most meaningful change in communities and people’s lives.
For additional information on where Charity Navigator is headed, please click here.
For further details on our new methodology for evaluating Results Reporting, please read the Results Reporting Concept Note.