Is it knowledge or data that is power?
One of the most common criticisms of the nonprofit sector is that it isn't data driven. Given the lack of data available and the cost to collect data, being data driven is often not a realistic expectation.
Alana Conner Snibbe, senior editor at the Stanford Social Innovation Review, gave a great presentation last year. She demonstrated that the measurements requested of grantees by foundations would typically cost $1 million to produce in a scientifically valid fashion. That is a big investment to secure a $25,000 grant for nonprofit with a $750,000 annual budget.Despite this overwhelming case, foundations continue to ask for data, and what they end up getting is faux data that is often unintentionally misleading. This not only undermines the trust between the foundation and nonprofit community, it also builds a culture that trades on poor data and half truths.
The president of a leading national nonprofit once told me that in the nonprofit sector - "knowledge is power." The reality is, it is data and not knowledge that is power. Sadly, most data is garbage and the opposite of knowledge.
Here is the most recent example I have seen of this trend: A recent report by a Boston nonprofit consulting firm showcases this headline as a key finding: "Fifty-eight percent of nonprofits surveyed are allocating 2% or less of their annual operating budget to support key functions."
Upon my first reading of this headline, I was shocked. Nearly 60% of nonprofit organizations are spending two percent or less of general operating budget to marketing, program operations, public relations, human resources, financial management, or technology? Wow. Nonprofit executives are truly miracle workers to be able to run their organizations with so little money for overhead.
Then I started doing the math. A basic financial audit (a de facto requirement for all nonprofits with budgets over $250,000) costs between 0.5% and 1% of your budget. That is just the audit - a nonprofit with a budget north of $1,000,000 will almost always have someone full time doing day-to-day accounting,operations and HR processing. Fully loaded that is a cost of about $75,000 (or 7.5% of the budget).
It wasn't adding up - there is no way that nonprofits spend that little on those six functions combined. I know their spending is thin, but not that bone thin.
Puzzled, I went back to the report and read the fine print (or at least non-headline print) and found this supporting statement: "The majority of survey respondents spent less than two percent of their operating budget in any one functional area." After reading this line a few times I realized that their finding was actually that 58% of the respondents reported that they spent less than 2% of their budgets in ONE of the six overhead functions.
Knowing that nonprofits spend very little money on public relations, one could conclude that 58% of nonprofits spend less than 2 percent of their budgets on public relations. Now that is not surprising and not really headline worthy. In many cases, I would be concerned if nonprofits were spending more money on public relations.
I then looked at their sample - 123 nonprofit respondents. We all see polls with +/- degrees of accuracy of 2-4% in the paper all the time. With this small a sample and considering they looked at nonprofits with budgets between $500,000 and $5,000,000 (10X) in three cities, the data likely has a +/- of close to 100%.
All that said, I will be the first to admit to spinning research results based on small samples. The goal here is not to point the finger at one report. The issue is more systemic. The desire to create data in the sector has led to far too many reports that would not stand up to scrutiny in any other sector.
How do we solve this problem?
For better or worse, any solution to an issue in the nonprofit sector needs to follow the money. Foundations funding a specific area (e.g. nonprofit consulting) should pool their funds and facilitate the creation of uber studies to benchmark the state of the issue and then create a common tool for evaluating success relative to other service providers.
In this case, foundations should fund a well designed survey with an meaningful sample that is based on input from the key players in the nonprofit consulting arena. This survey would look at the real state of the infrastructure in the sector and the key needs for consulting services.
They should then create a universal online survey tool for all nonprofit consulting clients to complete after projects. The data could then be sliced by the consulting firms and by the foundations. Ideally, it would also be made public to the nonprofit sector as a kind of "Consumer Reports" for nonprofit consulting services.
Until this happens, be sure to read all nonprofit data and reports with a critical eye - like you would read the results of the famous Pepsi Challenge of the 1980s.
Dirty data is not unique to non-profits, obviously. Governments are equally as guilty of not keeping good records, not dedicating staff-hours to data maintenance, and most gravely, not employing sound statistical analysis in working with and presenting data.
In order to motivate any organization -- governmental, private, or non-prof -- to change, you have to first answer the question, "what do they have to gain by better understanding their own numbers?"
In the case of these non-profits, maybe they discover that they have unused capacity, or that they're actually a lot more efficient in delivering services than they thought they were. And knowing these tidbits helps them improve their sales pitch. But there has to be some sort of core motivation to get people to move towards any system that includes more sound data analysis, because with numbers comes accountability. And ED's don't necessarily want that unless it's going to net their organization something positive, like a new grant or a new talking point about number of clients served.