Measuring Success: Do Your Metrics Tell Your Story?
Evaluation is clearly a sticky wicket. Now that we're smack dab in the middle of the information age, it's easier than ever for nonprofits to track all kinds of data that help them measure and evaluate their performance. We know down to the minute how many people are downloading reports, signing up to volunteer, sending emails to decision makers, visiting our clinics, etc. Pervasive Internet access and ever-shrinking hardware mean that we can collect and store more data than ever before. And we can publish that data more easily than ever before, increasing our transparency as we increase our measurements.
No one will argue that this is a bad thing. But are we really measuring what matters? Do all these numbers really tell us if we are meeting our missions? Take this number. Tell me what you think this number says about NTEN:
- 7158: September site visits to date at http://nten.org.
It does say a number of things. It says that we are less popular than Amazon.com and Facebook. It also says that we are not even beginning to scratch the surface of the number of nonprofits that could know about us.
But here's what it does not say:
- How engaged those 7158 people are with the NTEN site, or the organization;
- If those 7158 people think NTEN is worthwhile; and most importantly:
- Whether or not we are achieving our mission
I'm going to guess that this is the case for most nonprofits. Your web sites visits don't, by themselves, tell you anything about whether or not you are achieving your mission. You also don't learn if you're achieving your mission by tracking the number of clinic visits you get each year, or how many donors you have and what their average gift size is.
But you should be tracking these things. Because all together, they can tell you if you are, in fact, meeting your mission.
So let me paint a clearer picture of NTEN now. You can read about our mission in full, but in short, we create community to help our members find and learn about the tools, resources, and ideas they need to use technology to change the world.
Here are some of the stats we track to help us measure that mission:
> Site stats - nten.org (Year to date)
- 102,981 visits
- 62,292 Unique visitors
- 324,276 page views
- 3.15 pages per visit
- 3:12 average time on site
> Site stats - groups.nten.org (Year to date)
- 14,310 visits
- 8,581 unique visitors
- 101,469 page views
- 7.09 pages per visit
- 5:59 average time on site
- 5,070 profiles on the platform (3,367 at same time in 2006)
- 10,322 message posts
- 122 affinity groups
- 291 shared resources
NTEN Affinity Group Participants
> Membership: 2007 year to date (vs. same time period in 2006)
- Individual: 512 (427)
- Nonprofit: 464 (278)
- For-profit: 139 (144)
- Total Individuals receiving member benefits: 2783 (1936)
NTEN Membership Growth - 12 months
> Webinars: 2007 year to date (vs. same time period in 2006)
- Number of webinars: 24 (21)
- Attendees year to date: 732 (648)
> Newsletter: August 2007 (August 2006)
- Subscribers: 3873 (2673)
- Open: 1040 (950)
- Unique Clicks 394 (219)
NTEN Connect Newsletter Subscribers - 12 months
> NTC ATTENDEES:
- 2004: 644
- 2005: 793
- 2006: 812
- 2007: 1136
Looking at any one of these numbers alone -- especially without any of the trend information -- doesn't tell you very much about whether or not we are achieving our mission.
But together, they start to paint a picture. You can see the tremendous growth in our organization -- from members to newsletter subscribers to affinity group participants -- indicating that we are expanding our reach and aiding more nonprofits in service of our mission. You can see that more people are engaging in our programs -- clicking through to newsletter articles, attending webinars and the NTC, etc. -- indicating that we are providing better content to people, allowing them to learn more and do their jobs better, another key piece of our mission.
But all these numbers only get us part of the way there. How do I really know that NTEN is achieving its mission?
I get emails and phone calls and hallway conversations every day from people who want to share how an NTEN experience helped them learn or grow. And sure, I could add those to our giant spreadsheet of stats, but it would lose a lot in the translation.
At the end of the day, nonprofits are not about the number of widgets we move off the shelves. We are more than metrics. We are stories. We are the tales of the lives we touch and the communities we shape. And though metrics are a necessary and good part of our work, they are not the story.
Holly, this is one of the most practical pieces on website evaluation I've ever come across. Wish I'd written it myself. Thanks.
I've been advocating for more transparency from NTEN and we see the first halting attempt. One possible correction?
You claim 7158 September site visits and then go on to discuss them as 7158 people. I think you mean the number of unique visitors. It's probably less than 7158 but it's still probably in the 6900-7000 range.
It's clear the most important outcome in all these statistical measures is how many people are members of NTEN and how many people come to NTC. I encourage everyone I know who is a nptechie to join and go to NTC at least once. However, it's clear that site usage is a clear indicator of interest in NTEN when the NTC is not happening.
I still don't understand the reluctance to release real-time site statistics. We could easily compile much of the contextual information you cite in this post over time. Clearly, if the site stats start to dip that would probably indicate a lack of interest in NTEN. That's not good for you and for the nptech community and certainly not good for me and my blog.
I think people seem to think that there's a zero-sum game between myself and people who don't want to release site statistics. I've even been called, unkindly I think, a demagogue. This thinking goes: if Allan "wins", NTEN will lose. It's not a zero-sum game, it's a win for everyone if NTEN follows through on a commitment it makes to be accountable. If NTEN doesn't release site stats, that doesn't mean I "lose". It means that the community loses because now we can't use NTEN as an example to nonprofit execs about the power of openness and transparency.
I assure NTEN that the sky will not fall if the site statistics are made public. Pigs will not be seen flying over Central Park, snowballs will still be melting in hell and cats and dogs will still be fighting each other. Why not lead nonprofits and show them that a little bit of openness isn't dangerous? And that sometimes, as in real life, letting other folks start the conversation about you without having control over it is a Good Thing. After all, we're asking that of nonprofit execs every time the phrase Web 2.0 is mentioned in a new site redesign.
I have been following this debate and I have read all of Mr. Benamer's arguments. To be completely honest, I just do not understand why he is so passionate about web traffic stats transparency.
I am the Tech Manager for a statewide organization that works on access to health care. The Internet is a medium, a tool, with which we do our work. What happens between computers does not give a whole picture of our work--it is what happens between people that determines whether or not we are meeting our mission.
As Holly eloquently explains, web stats are one of many indicators that can help us understand what is happening between people.
Our *most* helpful indicator is the regular conversations we have with our constituents about how we help them, verbally and through regular surveys. We do share those survey results with our constituents and funders (and publicly, as appropriate).
We use surveys, verbal feedback, written feedback AND web stats to evaluate our usefulness and adjust our programming accordingly to stay on mission.
There are lots of good things to put energy into on the subject of orgs staying on mission and employing transparent feedback loops. I don't think sharing web stats is one of them.