If We Can Do It, So Can You: Mobile Evaluations at the 09NTC
Karl Hedstrom, NTEN
Every year at NTEN's annual Nonprofit Technology Conference (NTC), one of our major goals is to collect enough feedback from you, the attendees, to make our future conferences even better. Historically, we've accomplished this through the tried and true method of paper evaluations. At the 2009 NTC, we decided to give this system a major tech overhaul by implementing a mobile text messaging version of this: the Mobile Evaluation.
There were a number of reasons to make this change.
- First, paper evaluations use a LOT of paper (1,450 attendees x 6 sessions per attendee). Iin our continuing effort to make the NTC greener, saving paper is always a good thing.
- Second, the room setup required for paper evaluations has always been a huge undertaking for our limited staff (10+ minutes per room x 15+ rooms full of people networking with each other, all in less than 30 minutes), so anything to make this process quicker/easier was a plus.
- Finally, since the NTC is "the place to be" for nonprofits looking to put technology to use for their causes, we love trying out new tech tools for people to see in action.
After looking at our options, we partnered with Mobile Commons, who generously donated their system and expertise to us, and made the entire setup process a breeze. (I can't thank Matt Wilson enough for all his help.)
How it Worked
From the user's perspective, every session had a unique code (i.e. ntc101). To start an evaluation for a specific session, you would send a text message with that session's code to the phone number 69866; it was the same phone number for all our evaluations. The system then texted you back with the first survey question for that session, at which point you would reply with your answer. This would trigger the next question, and so on until the evaluation was complete.
The backend setup for all of this was surprisingly simple: We had a single phone number that all our evaluations would go through, and then a separate "campaign" was created for every evaluation. Each campaign was given a unique code -- used to match incoming text messages with the correct campaign -- and a unique set of responses/questions.
We ended up with about 90 different campaigns (one per session to be evaluated), and most consisted of the same 4 questions, plus a final thank you message. For the user, a single evaluation consisted of 5 sent and 5 received text messages.
What Worked Well
Overall, our move from paper evaluations to mobile evaluations was a huge success, and we definitely plan to stick with it for next year's NTC. It not only surpassed our expectations in regard to the three driving forces behind the change (less paper, less work, more tech), but it also didn't disappoint in perhaps the most important statistic for evaluation methods: the number of evaluations completed.
Paper use cut by 95%: Despite our new offerings of mobile evaluations (along with an online version), we still couldn't abandon paper evaluations completely, on the off chance anyone showed up at our technology conference without a cell phone or laptop. To accommodate these people, we ended up keeping a stack of about 15 half-page paper evaluations at the back of each session room, and then refreshed these stacks as they were used, going through roughly 30 half pages per room over the course of the conference. In 2008, when paper evaluations were the only option, we used approximately 50 full page surveys per session x 6 sessions, per room (so roughly 300 full pages per room over the course of the conference).
Staff time devoted to evaluations cut by 90%: With only 5 full-time NTEN staffers, there's never a shortage of things for us to be doing at the NTC. In previous years, setting up a room with paper evaluations would take approximately 10 minutes per room: collecting all the evaluations from previous sessions, setting out all the new evaluations, all the while keeping giant stacks of paper in order and trying to stay out of the way of people networking between sessions. This year, room setup was cut to about 1 minute per room: posting the new session code on the wall and refreshing the stack of paper evaluations at the back of the room.
Staff time spent consolidating data cut by 95%: With paper evaluations, it took a lot of time post conference to transfer all that information into a spreadsheet for analysis. This year, we still had a handful of paper evaluations to go through, but all the data that came from our mobile evaluations was already in a nice spreadsheet, ready for immediate analysis.
Evaluation completion rate wasn't horrible: This doesn't sound like high praise, but going into the conference, one of our main concerns was that people would be so unfamiliar with the new system that our completion rate would drop significantly. The fact that it actually increased slightly (from 871 to 900) was therefore a great success.
What Didn't Work Well
Not everything with our new mobile evaluation program went as smoothly as we would have liked, though most of the issues could have been minimized had we actually started implementing this system a bit more than a month before the NTC started.
Lack of buy-in and understanding by speakers: Our biggest allies in the effort to increase evaluation completion rates have always been the session speakers. We attempted to bring them on board at the last minute, but with so much else going on right before the NTC, our success was limited. While most speakers were at least aware of the new program, very few actively encouraged their audience to use it, while even fewer could answer any questions about how it worked.
Mobile codes were not widely available: In order for attendees to use the mobile evaluation system, they needed to know the correct code for the session they were evaluating. We made sure these were always displayed on the session room walls during the session, but if someone wanted to go back and evaluate a previous session, they were out of luck.
Cell phone coverage in the hotel wasn't great: This last one wasn't really our fault, but in some of the session rooms, the cell phone coverage was limited to nonexistent. This should have just been a minor inconvenience, but when combined with the fact that the session rooms were the only place people could obtain the session codes, it likely cut into our completion rates significantly.
Changes Planned for Next Year
Our biggest issue last year was obviously the last minute implementation and the resulting lack of exposure the program received. We're already taking the first steps to rectify that problem by coming up with a new plan of action now, nearly 9 months before the 2010 NTC.
Get early buy-in by speakers: We plan to introduce potential speakers to our mobile evaluation system right from the start. By next April at the conference, they'll be aware of and comfortable with the system. If all our speakers are able to push the system and answer questions about it, we should see a huge boost in evaluation completion numbers.
Integrate mobile codes with conference materials: Most of our conference materials are planned and completed months in advance of the actual conference. By starting on next year's mobile evaluation system at the same time, we should be able to fully integrate the mobile codes with our program book, signage, and online conference pages, making them much more visible and easier to find at the conference.
Introduction at the pre-conference community call: Every year, we have a series of pre-conference orientation calls with our community. This is where first-time NTC attendees can ask questions, and experienced vets can dish out advice. By introducing the mobile evaluation option during this call, we can build a lot of exposure and audience buy-in for the system. We may also consider trying some sort of pre-conference survey/giveaway using the system in order to get registrants comfortable with the format.
Fortunately, we still have a lot of time to figure everything out this year.
